This has been the first time for a long time where I've been chasing my own tail for hours on end. The Newsfeed simply would not load and I thought I had tried it all.
90% of all posts mention the need for the service running the User Profile Service App to have full control on itself via Permissions on the Service App in Central Admin. You'd think that is obvious. But nah! It apparently did not work for me.
So I went on a mission to demystify the distributed cache, appfabric and Newsfeed settings.
Without going into full detail on all the intricate details I found out along the way, here are my lessons learnt:
1) when uninstalling the appfabric, make sure to uninstall the hotfix first!
2) before re-installing the appfabric, ensure you remove that nasty PSModulePath Environment variable. Or the second install will fail over and over again.
3) when using Remove-SPDistributedCacheServiceInstance, wait some time before carrying on with anything.
4) The server which hosts the User Profile Service needs to also be a Distributed Cache host. Otherwise the User Profile Service Application - Feed Cache Repopulation Job will fail to connect to the cache host. So don't go removing that service from your app server quite yet...
5) If you end up with an invalid configuration and the Distributed Cache Service will not start via Central Admin, giving you a Cachehostinfo is null error, try this lovely nugget: http://rakasatria.wordpress.com/2013/02/04/sharepoint-2013-spdistributedcacheserviceinstance-cachehostinfo-is-null/ . Simple, effective and worked like a charm
6) Try Forcing the cache update via powershell: http://technet.microsoft.com/en-us/library/jj219560.aspx. The error message given here might just shed some more light on the problem.
7) And finally, check that you have actually given the right account the required permissions. In the end, it was the initial account permissions which were causing the problems, but I had mistaken the UPS Account and given the wrong account the permissions. And pretty much any other account under the moon, except for the one running the UPS.