Monday, October 6, 2014

SharePoint Online Powershell Connect-SPOService DTD Error

So you've been getting weird DTD error messages when trying to connect to your sharepoint online tenant using Powershell? well me too! and some Fiddling gave me the solution.

if you get a message along these lines:
Connect-SPOService : For security reasons DTD is prohibited in this XML document

then you're receiving some weird html instead of an authentication token back from SharePoint online. The easiest way to drill into this is to load Fiddler2 and track the traffic when you run the connection. Ignore the 403 and 401 errors. they are simply part of the normal authentication handshake. The interesting one is actually not the error but a 200 I found at the bottom of the list. a 200 code normally means all is good. but looking at it more closely I noticed that it did not come from SharePoint online but instead from Bigpond!! what? yeah. my ISP could not find the authentication endpoint under
hmpf. msoid? That's new to me. Looks like earlier this year MS introduced another DNS configuration step required for powershell commands to function:

notice the msoid CNAME entry? that's what I was missing. So as a quick fix I pinged, recorded the IP address that gave me back and entered a manual record in my hosts file. And voila, the connection works again! :-) Obviously I'll be updating my public DNS to match that CName.  but for now the quick fix did the trick!

hope this solves it for you too!

Tuesday, June 10, 2014

SharePoint LDAP apacheDS using ldapmembershipprovider

Hey folks,
It's been ages since I've written on my own blog and thought I'd share one of the funkier projects I've done recently.
Connect ApacheDS LDAP directory to a SharePoint 2013 server.
Simple, you might think. SharePoint supports LDAP. Doesn't it?

Well, there are two ways you can achieve this. Either by directly connecting the apache directory server to SharePoint or through ADFS. This will be a two post blog and in my first post I'll cover the simple, direct connection approach.

For SharePoint to understand LDAP users you need following ingredients
1) Claims Based Web Application. Luckily in SharePoint 2013, that's the default
2) LDAP Membership Provider & LDAP Role Provider
3) People Picker Magic
4) LDAP Provider that can provide the UPN in a way that SharePoint can handle it.

Luckily Claims is the default setting on SharePoint 2010 and above. So you'll have no problems ensuring a claims driven scenario. The tricky part is chosing your Claims Provider. But Luckily the built in LDAP provided by MS actually works! Just sadly, all documentation out there on using it assumes that you're connecting your Active Directory Domain as LDAP source. Why anybody would want to do that instead of just using the native windows integrated mode has always been a mystery to me. But we can use that provider to hook up our ApacheDS LDAP source. All you need to do is know your LDAP details. Here is a good technet article on configuring the LDAP web.config settings

Here is one that adapts to apacheds  

<add name="LdapMember"
type="Microsoft.Office.Server.Security.LdapMembershipProvider, Microsoft.Office.Server, Version=, Culture=neutral, PublicKeyToken=71e9bce111e9429c"
connectionPassword="pass@word1" />
<roleManager enabled="true" >
<add name="LdapRole"
type="Microsoft.Office.Server.Security.LdapRoleProvider, Microsoft.Office.Server, Version=, Culture=neutral, PublicKeyToken=71e9bce111e9429c"
scope="Subtree" />
The things to keep in mind with apacheds:
  1. Port is often 10389 instead of 389. so double check what port you're using, and pick the SSL/non SSL one respectively. (remember to set useSSL to true if you're going to)
  2. If you're going to go down the SSL path, make sure that the SharePoint server trusts the certificate.  
    1. open mmc on the SharePoint server
    2. Import the SSL cert used to secure the LDAP server into the trusted root authority for the local computer
  3. update your user and group containers respectively. (careful when testing with the built in admin account. that often sits in a different container alltogether and it's best you create a test account in your main OU)
  4. apacheDS expects the fully qualified DN for login purposes. so you need to fully qualify your connectionUsername when polling the LDAP repository
  5. Careful with those fully qualified user names. A cn can be very different to a uid!!! Look at the DN(distinguished Name) of your user account carefully. best try to copy it in its entirety.
  6. big Gatcha: apacheDS will send back the DN of the account it finds that matches. That's the fully qualified distinguished name. So when you go and give a user access you'll need to be able to check the return value against their DN. sadly the DN is not a queryable attribute by default. To make it queryable, you need to add the distinguishedName to your LDAP Schema. you can find it  by adding the extensibleObject class to your user schema. once you have the ability to add the distinguishedName atttribute to your users, and have set it to point at itself, the DN that the authentication module returns will match the distinguishedName attribute and voilla, the user will be let in.
  7. userName is often uid on an apacheds server, don't go looking for samAccountName
  8. group membership is often through the uniqueMember attribute on the group schema.
  9. Don't forget to configure your SecurityTokenServiceApplication!!! easily missed step
    1. Tip: don't just kill the clear tag which is killing the previous providers. That will come back to haunt you. do it properly and add the providers to the STSApp web config.
  10. Once you have ensured that it's all working, why not encrypt the web.config settings... you know, having plain text passwords in config files and that jazz.
    1. Just follow the above tutorial and replace connectionStrings with system.web/membership/providers (encrypting any part of the config file is easy once you understand that you need to treat it like xpath)
    2. do it again for system.web/roleManager/providers
  11. Oh, and you will want to add your LdapMember and LdapRole providers to your people picker. At least that's super simple. Just add to your peoplepickerwildcards section in the web config

some good resources on general SharePoint LDAP config using Forms Based Authentication (FBA) can be found here:

That's it! After you've spent many hours understanding the LDAP schema from apacheDS, tearing your hair out over distinguishedNames and got it all working for users to log in to SharePoint using their LDAP credentials you'll then be glad to know that FBA and Office Integration still sucks, even in 2013. The cookie that is used for the claim information can not be shared with the WebClient service which means that office and Explorer will ask for your credentials...again! Logging into different web applications (using different urls and thus cookies) will ask for credentials again! and best of all, if you followed the best practice recommendation from MS on how to setup your app framework, then you'll be hosting all your lovely apps on a completely different url and guess what, another login prompt again...

Now isn't that nice. But how can we provide a seamless single sign on experience to SharePoint for our LDAP users then? well FBA is not the way. but ADFS is! with ADFS you can configure all your webapps to use the same tokensigningauthority with the same realm and same claims. with ADFS tokens even explorer won't bug you for another login once you've logged into the website. A true single sign on experience!
But guess what. ADFS does not provide you with a way to authenticate users against anything other than Active Directory, you will need to add a ClaimProvider to your setup that can handle your LDAP login and create claims full of your lovely LDAP properties. But how? the answer lies in shibboleth.

So in Part 2 of this series I'll uncover the tricks required to get apacheds to talk to SharePoint using ADFS and shibboleth as the middle-men in the setup.
But before I go to the really cool ADFS bits, here some trouble shooting tips for the ldap connection

  1. Bump up your logging on the ldap server. Look for a file and change the logging level from WARN to DEBUG 
    1. After changing the logging, reset your apache service on the ldap server. if it fails to start again, then DEBUG was too much for it to handle. try setting an individual category instead of the root category to DEBUG.
    2. Always remember to reset the service or the new logging settings won't take effect
  2. install netmon and fiddler on all your machines to trace the handshakes going on.
  3. Bump up your SharePoint trace log  (found in Central Admin under Monitoring/Diagnostics)
  4. If the people picker works but sign in fails, double check your web.config for the STSApp. Often missed, especially when troubleshooting and not updating that one with the latest changes
  5. if you can't get the people picker to show users in Central Admin, remember to add the providers and peoplepicker settings to the web.config of CA as well!
  6. to save some pain consider updating the machine.config for the whole server. (plus the STSApp, which needs special treatment due to a very annoying clear directive
  7. Not all sections of the web config like being encrypted. You will need to be very specific about the section you encrypt. Don't try to go too broad.

    Monday, February 3, 2014

    SharePoint 2013 JSLink hide columns through calculated fields

    So you've probably come across heaps of cool articles that show you how to create fancy rendering solutions using JS Link Templates. But every example tends to deal with only one field (Template.Fields) or all fields (Template.Item)

    If you want to mess with more than one field it is really simple. Just add more to the array.
    Template.Fields = { "Budget" : {"View":somefunction}};
    Template.Fields = { "Budget" : {"View":somefunction},
    {"Timeline" : {"View":someotherfunction}}};

    Sometimes you want to use the value of a field without actually showing it as a column. For example when you're adding icons to other fields or enhancing a column with extra info.
    You'll notice is that the rendering template will fail if you do not include the desired field in your view, causing it to appear as extra annoying column.

    Here is a trick I stumbled across by accident:
    When you include a calculated column in your view, all referenced columns will become available to the current context through ctx.CurrentItem. So if you want to use the value of a field but don't want to include it as an extra column in the view, maybe you've got a calculated field you can abuse.
    In my case I had following calculated column
    BudgetStatus = [Actual Budget]/[Estimated Budget]
    That made all three fields available in my context.
    And now I wanted to include the Delivery Date for some funky rendering without making it a column? Simple. My new Budget Status looks like this:
    BudgetStatus = ([Actual Budget]/[Estimated Budget]) + ([Delivery Date] * 0)

    I know it's cheating. But hey, sometimes you have to cheat to get things done!

    Friday, September 27, 2013

    mouse pointer freezes in Windows 8

    Does your mouse pointer freeze in Windows 8 regularly? Alt-Ctrl-Del will get you to the lock screen where the mouse works fine, but once you're back on the desktop app your mouse won't move?

    Yup. happens to me regularly. Have not figured out what causes it, I'm guessing it has something to do with Outlook. or Remote desktops or Virtual Machines.
    Often the mouse will freeze when I switch into or out of a remote desktop or VM using Alt-Tab. and until recently I had not way bar reboot or try a million key combinations hoping they will release the lock. But through pure luck I have found a solution that works each time now :-)

    1) keep the Ctrl key pressed and your mouse will move. Like magic. release the key and the mouse freezes again. this at least will allow you to exit whatever focus you were in. For example exit the remote desktop console.
    2) Alt-tab to your running instance of Outlook. Each time I have done that, the lock is released and the mouse will behave normal again. Does not work with tabbing to other windows. But with Outlook every time.

    So is it Outlook who is locking the mouse? is it remote desktop? is it windows 8? I really have no clue. I'm just super happy I have found a solution which does not involve rebooting my machine.

    Monday, July 15, 2013

    Change Title to Filename in SharePoint 2013 Search Results

    In this post I will share a little secret on how to fix the Search results on SharePoint 2013.


    Whenever Search returns a Word result, it tries to be especially clever. If it prefers the first line of text in the document over the title field, it will use that instead of the title or filename in the search results. This is particularly annoying when the results are all forms and templates ,  where the first line in the document means absolutely nothing to the users.
    So how to fix this? After a long battle with Microsoft support I was told that this is by design.
    Some posts suggest not having a title at all. That way the filename will be chosen. Sadly that did not work in my case. It kept on returning the first line of the document. the only solution is to mess with the display templates used in Search. Luckily this is pretty simple nowadays.


    Open the Item_CommonItem_Body.html DisplayTemplate in SharePoint Designer (_catalogs/masterpage/Display Templates/Search)
    Not the Item_Word one!!! The Title logic is not defined here and any attempts to change it will be futile.

    on the commonitem body template add after line 21 following code

    if(ctx.CurrentItem.FileExtension == "doc" || ctx.CurrentItem.FileExtension == "docx"|| ctx.CurrentItem.FileExtension == "dot" || ctx.CurrentItem.FileExtension == "dotx"|| ctx.CurrentItem.FileExtension == "docm"){
      title =  ctx.CurrentItem.Path.replace(/^.*[\\\/]/, '');
    That will check for Word files, cut out the filename from the path using regex and replace the title with the filename.

    Hope this helps!


    Thursday, July 11, 2013

    Powershell fix for App master problem

    Ever come across this beauty?

    Sorry, something went wrong
    Accessing referenced file ...MasterPage.master... is not allowed because the reference is outside of the App Web.
    Oh, man this babe had me shouting and screaming for quite some time. But I now have a solution which will work every single time, even on Office 365!

    First, why is it happening in the first place?

    Three ways of causing it:
    1) you have a piece of code which applies a custom master page to your site. In a feature receiver for example. It iterates through all webs and updates the master reference. Sadly app webs, although on a completely different URL are still regarded a sub-web of your main site collection. And thus you'll change the masterpage reference of your app web and it will crash and burn as it is trying to access restricted resources outside of the allowed arena.
    2) you ticked the lovely check box(Reset all subsites to inherit this site master page setting) on the Masterpage admin page (ChangeSiteMasterPage.aspx)
     on a publishing enabled site collection which applies the master page to all sub sites. bang. apps get messed up.
    3) you deployed a design package. Bang. apps get messed up.

    Please keep in mind, that this problem only affects Apps which have an AppWeb. Ie. SharePoint hosted apps and Autohosted apps. Any App which does not need an app web will not be affected by this problem. Cloud Hosted Apps for example.

    How to avoid it?

    In your feature receiver, check isAppWeb on the SPWeb object before applying the masterpage. if isAppweb, then skip applying the new master page :-)

    But that won't stop the publishing feature to mess up your app web!

    Make a copy of the app.master (found in C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\TEMPLATE\GLOBAL), include it in a module in your app project and hard-code the masterpage references in the aspx pages. That way they won't take on the hand-me-downs from the publishing feature. Also lets you be creative with the master page and what to do with it...

     But what if you don't have control over the app code? I.e. added an app from the SharePoint 2013 app store?

    How to fix it?

    So, you can't avoid it and need to fix it? Easy :-) Well. took me some thinking and tinkering.
    You can't access the masterpage admin page on an app web :-(
    You can't connect to the app web using SharePoint Designer :-(
    But you can mess with it via Powershell :-)

    This is the script I came up with to fix up the master reference on an app web that got messed up:

    $siteurl = ''


       $site = Get-SPSite $siteurl
       $web = $site.OpenWeb('CorporateNewsApp');
       $web.CustomMasterUrl = "/CorporateNewsApp/_catalogs/masterpage/app.master"
       $web.MasterUrl = "/CorporateNewsApp/_catalogs/masterpage/app.master"
       Write-Host -ForegroundColor Red 'Error encountered when trying to fix app master url on ' $siteurl, ':' $Error[0].ToString();
    But what if you're using Office 365? Need to connect via SharePoint Online Management Shell. No Get-SPSite option available... or is there??? Just replace the urls, user and password and you're good to go.

    $programFiles = [environment]::getfolderpath("programfiles")
    add-type -Path $programFiles'\SharePoint Online Management Shell\Microsoft.Online.SharePoint.PowerShell\Microsoft.SharePoint.Client.dll'

       $siteurl = ''
       $username = ''
       $password = ConvertTo-SecureString -String 'PASSWORD' -AsPlainText -Force

       [Microsoft.SharePoint.Client.ClientContext]$cc = New-Object Microsoft.SharePoint.Client.ClientContext($siteurl)
       [Microsoft.SharePoint.Client.SharePointOnlineCredentials]$spocreds = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($username, $password)

       $cc.Credentials = $spocreds
       $site = $cc.Site;  
    $web = $site.OpenWeb('CorporateNewsApp');
       $web.CustomMasterUrl = "/CorporateNewsApp/_catalogs/masterpage/app.master"
       $web.MasterUrl = "/CorporateNewsApp/_catalogs/masterpage/app.master"
       Write-Host -ForegroundColor Red 'Error encountered when trying to fix app master url on ' $siteurl, ':' $Error[0].ToString();


    Hope this helps some of you out there with the buggy app model on office 365. I tried to get this resolved by MS Online Support but they proved completely incapable of solving this problem, let alone understand what I was on about in the first place.

    Thursday, June 6, 2013

    SharePoint 70-332 Exam Tips

    So a few weeks ago I passed the dreaded SharePoint 2013 Pro exam. And I must tell you, the panic was for naught. I expected it to be heaps tougher than the 331 predecessor. I was trying to read up as much as I could. Tried to prepare heaps. But in the end it had the same kind of questions as 70-331, just phrased differently with a more architectural spin on them.

    So if you sat and passed 70-331, my advice is to not freak out about 332 and sit very soon after passing 331. That way the studies you did are still fresh from the previous exam and will help you though this tough little cookie.

    Don't get me wrong, it was not easy. I found it as tough as the 331. It just was not any tougher and called upon the same set of knowledge which got me through 331.

    Hope this helps some of you aspiring to become the next MCSE SharePoint.

    For some tips on passing 70-331, check my earlier blog post