Tracking SharePoint User Properties with Microsoft Application Insights

Out-of-the-box page view tracking is dead simple to get working with every Web Analytic tool I’ve used, and Microsoft’s Application Insights available via the Azure platform is no different.

The Application Insights JavaScript code snippet is straightforward enough, albeit a little strange in that it is adding a script node to the DOM, but that’s fine:


var sdkInstance="appInsightsSDK";window[sdkInstance]="appInsights";var aiName=window[sdkInstance],aisdk=window[aiName]||function(e){function n(e){t[e]=function(){var n=arguments;t.queue.push(function(){t[e].apply(t,n)})}}var t={config:e};t.initialize=!0;var i=document,a=window;setTimeout(function(){var n=i.createElement("script");n.src=e.url||"https://az416426.vo.msecnd.net/scripts/b/ai.2.min.js",i.getElementsByTagName("script")[0].parentNode.appendChild(n)});try{t.cookie=i.cookie}catch(e){}t.queue=[],t.version=2;for(var r=["Event","PageView","Exception","Trace","DependencyData","Metric","PageViewPerformance"];r.length;)n("track"+r.pop());n("startTrackPage"),n("stopTrackPage");var s="Track"+r[0];if(n("start"+s),n("stop"+s),n("addTelemetryInitializer"),n("setAuthenticatedUserContext"),n("clearAuthenticatedUserContext"),n("flush"),t.SeverityLevel={Verbose:0,Information:1,Warning:2,Error:3,Critical:4},!(!0===e.disableExceptionTracking||e.extensionConfig&&e.extensionConfig.ApplicationInsightsAnalytics&&!0===e.extensionConfig.ApplicationInsightsAnalytics.disableExceptionTracking)){n("_"+(r="onerror"));var o=a[r];a[r]=function(e,n,i,a,s){var c=o&&o(e,n,i,a,s);return!0!==c&&t["_"+r]({message:e,url:n,lineNumber:i,columnNumber:a,error:s}),c},e.autoExceptionInstrumented=!0}return t}(
{
  instrumentationKey:""
}
);window[aiName]=aisdk,aisdk.queue&&0===aisdk.queue.length&&aisdk.trackPageView({});

This will get you tracking good metrics on any page view, which you can find best in the Logs section of the Application Insights “blade” in the Azure portal.

SharePoint scripting considerations

For SharePoint, if you want more, and you certainly do because the user data is very rich, it’s a little trickier to setup.

Script firing order is perhaps the least straightforward thing in SharePoint and perhaps this is changing in 2016(?), but usually, the developer is tasked with accounting for the disjoint manner in which scripts are loaded.

In most cases with SharePoint scripts, it’s basically a given that you’ll be layering your function calls under the arcane SP.SOD.executeFunc or SP.SOD.executeOrDelayUntilScriptLoaded functions to handle the unordered loading. For example:

SP.SOD.executeFunc('core.js', 'FollowingCallout', function() { FollowingCallout(); });

In the above, we use the script to declare a script dependency for the object that whats to be utilized in the subsequent function…MESSY!

SharePoint’s namespace challenge

The reason the code snippet is defining a script reference on the fly probably relates to the fact that it is trying to bypass the load order of other scripts and track the page view ASAP. But there’s a problem…

SharePoint uses a script dependency convention of namespaces which disallows JS objects to be defined if they already exist.

The problem for our script then is that both SharePoint and the script referenced by the tracking snippet define a Microsoft object, which is going to cause a collision if App Insights script does so first, and guess what, that’s what it’s trying to do.

The only way around this is to download the script being referenced in the snippet (in my case it is, https://az416426.vo.msecnd.net/scripts/b/ai.2.min.js) and modify the definition of the Microsoft object which occurs early on in the script…

Before (with Microsoft)


/*!
 * Application Insights JavaScript SDK - Web, 2.4.4
 * Copyright (c) Microsoft and contributors. All rights reserved.
 */
! function(e, t) {
    "object" == typeof exports && "undefined" != typeof module ? t(exports) : "function" == typeof define && define.amd ? define(["exports"], t) : t((e.Microsoft = e.Microsoft || {}, e.Microsoft.ApplicationInsights = {}))
}(this, function(e) {
// etc, etc...

After (changed to MSAzure)


/*!
 * Application Insights JavaScript SDK - Web, 2.4.4
 * Copyright (c) Microsoft and contributors. All rights reserved.
 * Modified by  on 
 */
! function(e, t) {
    "object" == typeof exports && "undefined" != typeof module ? t(exports) : "function" == typeof define && define.amd ? define(["exports"], t) : t((e.MSAzure = e.MSAzure || {}, e.MSAzure.ApplicationInsights = {}))
}(this, function(e) {

Then put a copy of the script on the SharePoint server. I recommend putting it under

<YOUR SITE>/SiteAssets/js

Next, change the tracking code snippet to refer to the modified, uploaded script.


var sdkInstance="appInsightsSDK";window[sdkInstance]="appInsights";var aiName=window[sdkInstance],aisdk=window[aiName]||function(e){function n(e){t[e]=function(){var n=arguments;t.queue.push(function(){t[e].apply(t,n)})}}var t={config:e};t.initialize=!0;var i=document,a=window;setTimeout(function(){var n=i.createElement("script");n.src=e.url||"https://yourserver.sharepoint.com/SiteAssets/js/ModifiedAppInsightsScript.js",i.getElementsByTagName("script")[0].parentNode.appendChild(n)});try{t.cookie=i.cookie}catch(e){}t.queue=[],t.version=2;for(var r=["Event","PageView","Exception","Trace","DependencyData","Metric","PageViewPerformance"];r.length;)n("track"+r.pop());n("startTrackPage"),n("stopTrackPage");var s="Track"+r[0];if(n("start"+s),n("stop"+s),n("addTelemetryInitializer"),n("setAuthenticatedUserContext"),n("clearAuthenticatedUserContext"),n("flush"),t.SeverityLevel={Verbose:0,Information:1,Warning:2,Error:3,Critical:4},!(!0===e.disableExceptionTracking||e.extensionConfig&&e.extensionConfig.ApplicationInsightsAnalytics&&!0===e.extensionConfig.ApplicationInsightsAnalytics.disableExceptionTracking)){n("_"+(r="onerror"));var o=a[r];a[r]=function(e,n,i,a,s){var c=o&&o(e,n,i,a,s);return!0!==c&&t["_"+r]({message:e,url:n,lineNumber:i,columnNumber:a,error:s}),c},e.autoExceptionInstrumented=!0}return t}(
{
  instrumentationKey:""
}
);window[aiName]=aisdk,aisdk.queue&&0===aisdk.queue.length&&aisdk.trackPageView({});

Getting and tracking SP User Properties

There’s a way around all this load-order chaos, you can track user properties along with page views. It can be accomplished with a call to SharePoint API endpoint GetMyProperties, with that response then provided to the trackPageView function. The endpoint can be accessed at the following address (keeping in mind that subsites may precede the /api/ part):

/_api/SP.UserProfiles.PeopleManager/GetMyProperties 

The following code snippet makes use of the Azure documentation sample, but with the following modification: I remove the page track call, and put it inside the return callback for a request to SharePoint API endpoint GetMyProperties.


var sdkInstance="appInsightsSDK";window[sdkInstance]="appInsights";var aiName=window[sdkInstance],aisdk=window[aiName]||function(e){function n(e){t[e]=function(){var n=arguments;t.queue.push(function(){t[e].apply(t,n)})}}var t={config:e};t.initialize=!0;var i=document,a=window;setTimeout(function(){var n=i.createElement("script");n.src=e.url||"https://yourserver.sharepoint.com/SiteAssets/js/ModifiedAppInsightsScript.js",i.getElementsByTagName("script")[0].parentNode.appendChild(n)});try{t.cookie=i.cookie}catch(e){}t.queue=[],t.version=2;for(var r=["Event","PageView","Exception","Trace","DependencyData","Metric","PageViewPerformance"];r.length;)n("track"+r.pop());n("startTrackPage"),n("stopTrackPage");var s="Track"+r[0];if(n("start"+s),n("stop"+s),n("addTelemetryInitializer"),n("setAuthenticatedUserContext"),n("clearAuthenticatedUserContext"),n("flush"),t.SeverityLevel={Verbose:0,Information:1,Warning:2,Error:3,Critical:4},!(!0===e.disableExceptionTracking||e.extensionConfig&&e.extensionConfig.ApplicationInsightsAnalytics&&!0===e.extensionConfig.ApplicationInsightsAnalytics.disableExceptionTracking)){n(""+(r="onerror"));var o=a[r];a[r]=function(e,n,i,a,s){var c=o&&o(e,n,i,a,s);return!0!==c&&t""+r,c},e.autoExceptionInstrumented=!0}return t}({
    instrumentationKey:""
});

window[aiName]=aisdk,aisdk.queue&&0===aisdk.queue.length;

jQuery.ajax({
    url: "/_api/SP.UserProfiles.PeopleManager/GetMyProperties",
     type: "GET",
    headers: { "accept": "application/json;odata=verbose" },
    success: function(data){
        var userinfo = {};
        userinfo["Department"] = true;
        userinfo["Office"] = true;
        var properties = data.d;
        if(properties.UserProfileProperties.results != null && properties.UserProfileProperties.results.length > 0)
        {
            for(var p in properties.UserProfileProperties.results)
            {
                if(userinfo[properties.UserProfileProperties.results[p].Key])
                userinfo[properties.UserProfileProperties.results[p].Key] = properties.UserProfileProperties.results[p].Value;
            }
        }
        appInsights.trackPageView({name:document.title, title:window.location.href, properties:userinfo});
    },
    error: function (xhr) { 
        console.error(xhr.status + ': ' + xhr.statusText); 
    } //End error method
}); // end of Ajax call to GetMyProperties

…This approach does not depend on SharePoint script loading order. Even the jQuery use not necessary since it can just as well be a standard JavaScript XmlHttpRequest.

In the above code, the desired fields are called out, rather than simply copying all user fields (which you could do), but the point is that you usually don’t want to track personally-identifiable info (PII) with web stats. So in the above snippet, I’m tracking simple the department and the office location of the user by iterating through all available properties and copying those values out.

I recommend doing this even in you want PII because there can be a lot of useless junk in the user properties, and App Insights does change by data volume.

Which brings me to my final point…

What about cost?

A lot of people go with Google Analytics because you can do all this and it’s free. However, GA doesn’t offer all the capabilites of Application Insights, so that’s something to consider. Furthermore, a lot of times companies avoid Google services simply because they have a, shall we say, dodgy reputation regarding intellectual property. So companies will go with Adobe Analytics, WebTrends, or other offerings. And Microsoft’s Application Insights is a contender here.

With Azure, nearly every resource you spin up now has a fee attached. Currently App Insights presently charges $2.30/GB of tracked data.
The total cost is depends on the amount of website traffic you get and the content load of your SP site. And typically, SharePoint has a ton of fluffy libraries to load.

I’m going to call a medium trafficked site is one that sees about 1000 visits/day. Let’s pad that figure by rounding up to $3 for easy math, and 30 days in a month; so we have $3 * 30 * 12 = $1080/year using Application Insights.

That’s certainly competitive compared to other platforms. But it should be noted that the reporting aspect of Application Insights has more of a learning curve and probably wants a customer engagement to get a decent dashboard setup for analysis.

Every issue, quirk, and glitch I ran into while developing a SharePoint Add-In

My job can be heavy with the SharePoint Online, and I do like to tinker around on it using the REST API and Script Editor web parts, and you can do cool stuff just with those.

But as soon as you start specking out workflows, it’s probably time to elevate to Apps or Add-Ins.

I can’t figure out what SharePoint wants to call them, but when you go to develop it in Visual Studio, you’ll be picking the SharePoint add-in project:

sharepoint-addin-project.jpg

I set out to create a polling add-in because I already had it working as a REST API script web part, so I figured, why not package the thing?

But the simplicity with which you can build out your app via Visual Studio’s bountiful UI is almost wholly negated by the weird ticks will will encounter as you develop.

I will try to catalog them here…

Extra, un-asked for Features

The first issue I ran into was new Feature nodes would be added depending on what I deleted from and added to the project. Visual Studio would seem to lose track of the Feature1 that it created by default, and begin to add new items to a second or even 3rd Feature node.

When that happens, delete the extra feature nodes, and take a look at Feature1 to be sure it is including all the relevant items of your project (which is probably all of them).

sharepoint-addin-quirk-extra-features

Retraction, Retraction, what’s your function?

When you re-deploy your add-in, it sometimes needs to retract the one that’s already there, and many issues arise.

One is, retraction can dodder on indefinitely, causing you to need to cancel the operation (the option should appear under the Build menu). But then you will need to rerun retraction, which is fine, except when SharePoint thinks the add-in is retracted.

A couple times I’ve had to go onto the deployment site, and delete the add-in, but even when cleared from the recycle bin and second stage recycle bin, SharePoint still believes it’s there!

One solution I found is to tick up the version (after you’re certain the add-in is removed), this way, SharePoint will not insist the app is there with the same version, because you changed the version:

pollster-app-manifest

(h/t this thread)

Confused schema

It’s not enough to have lists with separate schema settings, if their Type field is the same, SharePoint may confuse the schema, and you will see two lists with the same field set, which is jarring, really. Even if you completely delete the list, and recreate it, remapping every association it had in the project, SharePoint may find the schemas of the second list confusing.

One user found the solution was to set the type of the second list to 1000, and tick up as you add even more lists with possibly confusion inducing schemas.

(h/t this thread)

List Field Default Values

It’s not that there’s an issue with setting default values for your list fields, its just that when the UI is so comprehensive as it is in Visual Studio, it can be perplexing to need to do something on the code-level, especially setting a default value of a field which is standard procedure.

To do this, you have to edit the schema of your list, and add a sub-element <Default> within the <Field> element:

sharepoint-addin-quirk-default-list-field-values.jpg

(h/t to this thread)

Hiding list fields from the New, Edit, and View forms

Not every field of your list should be shown on every form, unless you want to give the user that sense of vague formlessness so common in modern art today. But to hide the field, you have to specify a series of values in the list Schema.xml.

Go tho the <Fields> element, and find the fields within that you want to show or hide, by adding the following (True/False) properties:

  • ShowInNewForm
  • ShowInEditForm
  • ShowInViewForms

 

HOW DO I GET THE STUPID SITE PATH TO MY ADD-IN?!

Alas, there are many answers to this on various developer forums, and none of them work.

Basically, your add-in is going to situate itself in an iframe that loads content which is outside of the site it gets added to, so it’s not really aware of some key globals such as _spPageContextInfo, or SP.ClientContext.get_current(), and you can’t add scripts to fix the problem because ultimately…it’s in a separate place.

NOTE: If you need to talk to info that resides on the destination site, your going to need to use SharePoint’s cross-domain ajax functionality, which was beyond my purpose, but you can read about it here.

If you’re simply looking to using SharePoint REST API within your web part’s quarantined site/section/ward, you need to pass in that URL via the web part parameters. SharePoint has a bunch of dynamic parameters called tokens, listed in the second table (Tokens that can be used inside a URL) here, but you have to add them.

sharepoint-addin-quirk-passing-the-app-path.jpg

Once you add them, you can build the REST API calls off of them, especially the {AppWebUrl} parameter. I suppose you could do some text processing on the document.URL string to get this value, but that seems risky, and SharePoint ought to tell you where it put your add-in.

If you’re interested in making calls to the site you’re adding to, you’ll want to make use of the {HostUrl} parameter. This may require some permissions setup in the AppManifest, but that’s further than I went.

Showing a list on your Add-In’s default page

My add-in makes use of a couple of lists, one of which I want to show on that default page, which I figure should be the sort of administrative front-end of the app and provide the user with some web parts to let them perform basic content addition, but don’t expect SharePoint to just give one to you.

You have to edit the elements.xml of the Pages node in the project, and add the following within the Default.aspx page listing:

<?xml version="1.0" encoding="utf-8"?>
<Elements xmlns="http://schemas.microsoft.com/sharepoint/">
  <Module Name="Pages">
    <File Path="Pages\Default.aspx" Url="Pages/Default.aspx" ReplaceContent="TRUE">
      <AllUsersWebPart WebPartZoneID="HomePage1" WebPartOrder="1">
        <![CDATA[
          <webParts>
            <webPart xmlns="http://schemas.microsoft.com/WebPart/v3">
              <metaData>
                <type name="Microsoft.SharePoint.WebPartPages.XsltListViewWebPart, Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" />
              </metaData>
              <data>
                <properties>
                  <property name="ListUrl">Lists/Polls</property>
                  <property name="IsIncluded">True</property>
                  <property name="NoDefaultStyle">True</property>
                  <property name="Title">Polls</property>
                  <property name="PageType">PAGE_NORMALVIEW</property>
                  <property name="Default">False</property>
                  <property name="ViewContentTypeId">0x</property>
                </properties>
              </data>
            </webPart>
          </webParts>
        ]]>
      </AllUsersWebPart>
    </File>
 </Elements>

Now, in the Default.aspx page itself, you can add markup that refers to this element data you just added:

sharepoint-addin-quirk-default-page-webpart

Wait, what’s wrong? You mean you want to see more than just the title field of your list in that web part?

I’m not 100% on this part, but this change worked for me:

Edit the <ViewFields> tag in the Schema.xml of your list where you find a <View> element with BaseViewId=”0″, Type=”HTML”…

sharepoint-addin-view-fields.jpg

Taking Add-In to the Market

I wanted to share, and possible earn some coin, off my frustration developing this supposedly simplistic add-in, but in doing so, I hit the following wierdness:

You’ve got to select at least one Supported Locale in the AppManifest, which was fine with me, I just wanted English to start, but if you just use the UI, it won’t add the setting properly. You need to go into the AppManifest.xml code-view and edit the xml so that it’s en-US (xx-XX).

sharepoint-addin-quirk-locale.jpg

Beyond that, the screenshots simply must be 1366px X 768px, naturally! Which seems a little unfair to little Add-Ins like mine which I had to pad-out with blank space because, seriously, it must be this size.

How to pull an Instagram feed into Sharepoint

Naturally, the cross-domain ajax call is going to be a problem, and SharePoint’s library adds a lot of unnecessary complexity to what would ideally be a simple call.

The key is to use JSONP and jQuery; you provide a callback function to the Instagram endpoint, and then specify the datatype “jsonp” in the ajax call, and jQuery will handle the invocation of the callback when the data is ready.

On the Instagram side, you don’t even need to step outside the client’s sandbox mode.

In Instagram, you’ll need to go to the Developer section and register a new client.

instagram-register-new-client-id

You’ll need to uncheck the Disable implicit OAuth:

instagram-register-disable-implicit-oauth

The new client will appear under the Manage Clients section. Copy the Client ID from the listing.

Then, under the security tab, provide a redirect address. This can be anything because we will not be leaving sandbox mode to achieve the connection, it will simply be any valid address Instagram redirects to and at the same time provides a token value at the end of that URL.

I’m just going to use brave.com because I like the browser and I don’t like Google:

instagram-redirect-urls

Now you need to hit the Instagram authentication URL, which is:

https://www.instagram.com/oauth/authorize/?client_id=[CLIENTID]&redirect_uri=[REDIRECTURL]&response_type=token&scope=public_content

Replace the Client ID with the value you copied off the client listing, and replace the redirect URL with exactly what you entered above (so if there is a ‘/’ at the end of the URL you provided, you will need to enter that).

After you modify the URL, and enter it into the browser and go, you will be prompted to confirm access for your client, when you do so, you will be taken to the redirect site you specified and the access token will appear at the end of the URL, like so:

https://brave.com#access_token=123456789.ab22100.34a13013341345283a641139940c38f8

Copy the value which appears after “access_token=” and use it in your api calls:

$.ajax({
  url:"https://api.instagram.com/v1/users/self/media/recent/",
  data:{access_token:"123456789.ab22100.34a13013341345283a641139940c38f8"},
  type:"GET",
  dataType:"jsonp",
  jsonpCallback: "handleInstagramResults"
  });

…this call will return JSON of the most recent posts, and jQuery will invoke handleInstagramResults to process it.

Notice I’m using the /self/ endpoint above; this doesn’t require that the app be migrated out of the Sandbox Mode or a user ID.