< Go Back to Homepage

SharePoint: Only request data that you really need

by Andreas Grabner, Jan 13, 09

One of the main performance problems that we can witness in any type of application that has to deal with data from a database is that too much data is requested. Requesting more information than actually needed by the current use case on hand results in additional

  • query overhead in the database to gather the requested information
  • communication overhead between the database and the application
  • memory overhead on both the database and the application

Scenario 3: Use SPQuery and SPView to only request data that you need for your use case

Looking back at the previous two posts (Scenario 1, Scenario 2) you can see that the executed SQL Statement always selected ALL items from requested SharePoint list. You can tell that by looking at the SELECT clause which says: SELECT TOP 2147483648 …

Limiting the number of returned rows

In case you only want a limited result set when accessing items in a SharePoint list you can make use of the SPQuery.RowLimit property. Here is an example:

SPQuery query = new SPQuery();
query.RowLimit = 100;
SPListItemCollection items = SPContext.Current.List.GetItems(query);
for (int itemIx=0;itemIx<items.Count;itemIx++) {
  SPListItem listItem = items[itemIx];
}

Using the SPQuery object with SPList.GetItems will result in the following SELECT clause:

SPQuery.RowLimit limits the number of records retrieved from the SharePoint Content Database

SPQuery.RowLimit limits the number of records retrieved from the SharePoint Content Database

In the previous example we already limited the number of items that we want to retrieve. We however still request ALL columns that are defined in the SharePoint list. This might be ok in case we really need all columns to display to the end user or if we need all of them to perform some calculations. In most of the cases we however only need a few and not all.

Limiting the retrieved columns

There are two ways of limiting the columns to retrieve from the database:

The sample code from above can therefore be changed in the following two ways:

SPQuery query = new SPQuery(SPContext.Current.CurrentView.View);

or

SPQuery query = new SPQuery();
query.ViewFields = "<FieldRef Name='ID'/><FieldRef Name='Text Field'/><FieldRef Name='XYZ'/>";

In both scenarios the SELECT clause will only contain those fields that are defined in the SharePoint view respectively those fields that are referenced in the ViewFields property. The following illustration shows the difference in the SELECT clause:

SELECT clause only selects those columns defined in SPView or ViewFields

SELECT clause only selects those columns defined in SPView or ViewFields

Conclusion

There are different ways to retrieve list data with the SharePoint Object Model. It provides mechanisms to specifically query the data that you really need in your use case. Its important to understand that the default access methods always return ALL rows and ALL columns of the underlying SharePoint list. Make use of SPView’s and SPQuery’s to filter the data that you need already on the database layer in order to reduce overall overhead and therefore improve performance and scalability of your SharePoint application.

Update: Read the summary blog of my blog series about The Top SharePoint Performance Problems. As SharePoint is built on the .NET Platform you might also be interested in my latest White Papers about Continuous Application Performance for Enterprise .NET Systems

Share
  • Facebook
  • Reddit
  • Digg
  • del.icio.us
  • StumbleUpon
  • LinkedIn
  • MisterWong
  • Technorati
  • E-mail this story to a friend!
  • Google Bookmarks
  • DZone
  • FriendFeed
  • Ping.fm
  • Slashdot
  • Twitter

Related posts:

  1. SharePoint: Page through SharePoint lists SharePoint lists can contain thousands of items. We have all heard...
  2. How to avoid the Top 5 SharePoint Performance Mistakes SharePoint is without question a fast-growing platform and Microsoft is...
  3. SharePoint: List Performance – How list column indices really work under the hood Have you ever wondered what is really going on under...
  4. SharePoint: Lookup value Performance In SharePoint you can define lookup columns in your lists....
  5. Performance Considerations when using SharePoint Object Model The SharePoint Object Model allows external applications or hosted WebParts...

Trackback

9 comments yet

  1. As you know this is a topic that I talk and blog about often so I appreciate all efforts to bring this information to people’s attention. I should point out, however, that ViewFields limits the amount of data returned but does not, in fact, decrease the overhead of the associated query. The real problem here is the underlying table structure; no matter how many fields you actually require in your return data set, the query still has to iterate every item in every list in the content database as ALL list items are stored in the UserInfo table. Limiting the amount of data returned simply decreases the payload of the returned query, which is minimal in most cases. True performance gains can only be achieved with a restructuring of the underlying relationships, which is something that is not likely to happen anytime soon. Until then, we are limited to working around these limitations by retrieving items into memory and using CAML, PortalSiteMapProvider, Search and web services.

    Also, the View functionality can be simulated by making a web services call that specifies the return fields without having to first create a view in the target list. This is a more flexible approach as the developer can specify exactly what they need within their data access assembly and not be limited by views defined in the UI. Refer to the GetListItems method of the Lists web service for more information: http://msdn.microsoft.com/en-us/library/lists.lists.getlistitems.aspx.

    Good post. Keep ‘em coming!

  2. Thanks for your feedback.
    You are right – limiting the the ViewFields doesn’t limit the number of items that we have to iterate – but – it decreases the result set in size. It should lower the overhead on the database, lower traffic on the network and lower memory consumption in the application as less field values must be kept in memory.

    I am going to continue posting topics about the SharePoint Object Model and what is really going on under the hood.

  3. Lars Nielsen @ 2009-03-20 10:58

    Interesting post thanks, especially taken together with Eric’s reply. Out of interest how did you manage to extract the SQL queries?

  4. dynaTrace’s PurePath Technology allows us to capture contextual information along the execution path of a single transaction/request. This context information can be http headers, parameters, method arguments, sql statements, bind variables, …

    With the PurePath Technology we are able to trace every single transaction that is executed in your system – whether it be in a live production system, a system used for load and performance testing, your continuous integration environment where you execute your unit tests or on your developers machine. For more details see http://www.dynatrace.com/en/PurePath-transaction-tracing.aspx

  5. rivendell @ 2009-11-17 21:08

    When I attempt to use the SPQuery to apply a custom filter I’ve written on all the events within a view, I end up with the custom filter being applied to the EVENT list and not the VIEW list.

    Here is my code…

    SPQuery query = new SPQuery(listview);
    query.CalendarDate = webPart.CalendarDate;
    query.Query = fullCamlQuery; //caml query to filter on events within the view
    query.ExpandRecurrence = true;
    SPListItemCollection collListItems = eventsList.GetItems(query);

    Any suggestions on how I can get my custom caml query filter applied on the view and not the event list?

  6. I am not sure if I fully understand your question but what you have to keep in mind is that a view is basically nothing else then a filter on a list. When you use an SPQuery you can use it in two ways
    a) use the filter defined by a View
    b) define the filter with CAML

    If you want to combine the filter settings of an existing View and your custom CAML query I believe you need to create a CAML query that combines both filter settings into a single CAML query.

    In your case I believe that setting the Query Property overwrites the filter settings that you passed to SPQuery in the constructor.

    Makes sense?

  7. rivendell @ 2009-11-18 04:02

    Thanks a lot for the feedback, you were right about using a query that combines both the filter settings of the view and the CAML query; I think this is why I was having so much trouble. So I went with just option (b) and defined the filter via the CAML query and it worked great.

    Thanks!

  8. Is there much difference between the queries like
    SPContext.Current.Web.Lists["MeetingList"] and “<FieldRef Na…"?

  9. There is a huge difference in using an SPQuery object and directly accessing the SPList. With SPQuery you have the chance to narrow down the items that you want to have and also the columns that you want to have – causing less stress on the database.
    So – if you can – use SPQuery and specify what data you really need.

Add your comment now