I’m Done – Behemoth Post / Explanation / Conclusions

Before I left on my vacation two weeks ago, I was starting to get more inspired about doing more with this blog. I joined twitter, I starting putting together technical content to post…
But  I was also getting tired of SharePoint, as previous posts show. I wanted to take my week offline and see if that refreshed me.

It did the opposite. I’m less inspired by SharePoint than I ever was.

So this blog is complete. I think there is enough material here that it has served it purpose. It really doesn’t add anything to my personal or professional life, so here is one big final post to close things out. I have heard from some people recently regarding my Domino work… I’ll get back in touch with you folk via email when the time is right. Not that you will know it is me.

Was/Is SharePoint a success?

Partially. We had many issues when we first rolled out our portals. The Server farms were badly implemented, the UI designs were horrid, they were thrown together badly incurring massive technical debt, and massive maintenance costs. That is where I came in. Since then, we have restructured the farm, and while that effort still has many tasks before complete, we are stable with a plan to get to our ideal infrastructure within the next 6 months. We have refactored the UI designs, and their tehcnical implementation, leaving the content publishers with as much flexibility and power as any content management system. Our collaboration areas have grown, and our management of them has evolved, so we are in good shape for oganizational collaboration.

However, the pains at the beginning mean that adoption of the technology was below expectations. Some groups gain much benefit from SharePoint, others ignore it or are downright annoyed by it. It was not the business-changing technology that was hoped for, nor did it save costs.

What would I do differently if I could go back and start from scratch.

1) Don’t use SharePoint. Sorry, but the cost of implementation and support is high, and I think even a perfect implementation would offer a poor ROI. But ignoring that…

2) Don’t skimp on technical planning. Think very carefully about long-term maintenance of the platform, and dig deep when analyzing the impacts of your decisions. Make every manager do a google search on “technical debt”, and spend at least 30 minutes following the results.

3) Don’t hire consultants — at least, not when we did, when MOSS 2007 was so new. At the time of our launch, nobody was a MOSS 2007 expert. So why pay consultant rates for people who basically are using us to learn a new technology, and then just writing .NET code anyway. Instead, hire some really good, sharp people internally, and let them run with it.

4) Don’t fall behind on patches. Ever. Microsoft knows that SharePoint has issues, and actively tries to fix them. If you are going to marry yourself to Microsoft, make it a production relationship, getting their latest patches and fixes, and keeping in communication with them for support and to know their plans.

5) Train your users. Repeatedly. Record the trainings. Make them available online. The more powerful your users are, the more time the IT staff can spend on the platform itself, or on custom code. But that power needs to come with knowledge and insights that only training, experience, and a solid support staff can provide. Put in the effort to give your community those things…

6) Do not accept mediocrity in your IT staff. We all know that the corporate world has more than its share of mediocre IT staff. There are reasons for this, and it isn’t a bad thing in many cases — Not everyone can be a superstar, and much of corporate IT doesn’t need superstars. But SharePoint does. At least as of today – I have hopes that in 3-5 years, the platform will mature into something more friendly, stable, and robust. But for now, it is plagued with problems. Any IT staff member can handle these problems in the short term. But it takes top talent to do so quickly, repeatedly, and often, for months and years on end without running into morale issues. Running a SharePoint implementation is the 100-mile run of IT. Sure, it can be done. But the people who can do it well are freaking machines, not your average IT guy. Get those people on board Day 1 if you want SharePoint to truly improve your business.

7) Code Reviews and Architecture reviews. Do them. Mistakes in design become nightmares in support. Trust No One.

So where does our Notes Migration Stand?

Over the last 18 months, I reduced our Lotus Notes environment from 6000 applications to just over 150. 35 of those are group mailboxes that are in the process of moving to Exchange. 50 of those are basic logs of what happens in our business — the SharePoint replacement is in a pilot program, and they should be gone in a matter of weeks. 50 more  are minor apps, and would only take about 3-6 months to move, but we put that effort on hold due to the recession and its impact on our business. Our 5-10 major, massive apps are slated to be replaced by 3rd party products. Again, plans are in place, but the recession has those projects on hold.

In short, we expect to have moderate usage of around 50-65 apps for 2009, with everything else being shut off with a resonable amount of effort in 2010 or 2011, depending on just how long this recession lasts.

The migration wasn’t rocket science. A lot of it was just being willing to dig in and do the work. It does require an understanding of the business, the usage of the existing platform. It also takes effective communication and negotiation skills. A consultant would not succeed by wlaking in the door, getting specs, and making plans nearly as well as a long-time business analyst who understands not only what goes on, but why. The migration requires less technical skill, and more business comprehension.

Recession?

Honestly, without the recession, I’d probably keep going with the blog, the migration, and probably enjoy it more. But being shut down into a maintenance-only mode means that there won’t be much to share anyway. Time to move my personal efforts into other projects…

SharePoint  Conclusions

1) If you are already a solid Microsoft shop… sure, go for SharePoint. You probably have the talent to make it work. If not, don’t move to a Microsoft-based platform just for SharePoint. There are other alternatives that will serve you better.

2) Don’t buy into the marketing. SharePoint is an effective collaboration tool. But nothing more. Use its strengths, but don’t expect it to transform your business in any way.

3) Everyone should revisit their stance on SharePoint in 3-5 years. Us Notes folk benefitted from the early leadership of Ray Ozzie. I believe SharePoint will, too. If you already run it, look for future enhancements. If you choose not to run it, don’t let today’s SharePoint deter you from the SharePoint of tomorrow. I think it may come together.

Lotus Notes Conclusions

If you’ve got a Notes/Domino platform running, stick with it. If you don’t like it for some reason, fix the problems with your implementation of Notes. Don’t throw it out, expecting a new platform to be better.

Personal Conclusions

1) SharePoint isn’t fun.

2) SharePoint pays well. You earn every penny. Don’t go into it for the money. Be a fan of the Microsoft Kool-Aid or it will slowly inceinerate you from the inside out.

3) The breadth of my IT experience serves me very well. I’ve been a programmer, and analyst, a trainer, a helpdesk grunt, a server admin, a technical writer, a web designer, and a manager. I have used every single thing that I learned in all of those disciplines over the last 18 months.

5) I have learned SharePoint. I have a solid bullet point on my resume and in my skill set. I don’t really care. Kinda like my Notes admin skills — sure, I can do the job, but I’ll likely never apply for that job. The SharePoint work has done well for my resume, salary, job stability, and all those things that I really should care about. But I’m a programmer at heart – none of that means much to me when the platform itself is dreary. Don’t get me wrong — I fully appreciate being blessed with a stable job at this point in time.But inspiring, it is not.

6) The past 18 months have been frustrating and educational, but with moments of great satisfaction when we finally do get things running the way we want.

7) In short, I’m getting too old for this crap.

Nevertheless, the closure of this blog changes nothing for me personally. I have the same job, the same roles, and the migration and SharePoint work will continue in my day-to-day life.

Final Words…

I intend to leave this blog up, as it does get search traffic, and I hope my old content can help some people out.
The gmail address is forwarding to my personal address. If anyone has reason to get in touch, feel free… I’ll be reachable in that way for the foreseeable future.

Thanks for reading and participating in the blog…

jQuery Performance: Problem & Solution

One of my current tasks is to refactor our SharePoint portal’s design.

The original design was done with heavily laden with tables and large images.And each new subsite had its own master page, style sheet, and images directory. It was performing poorly in the browsers, and making it painful to create new subsites.

So I’m making a CSS-based version, removing all the images, simplifying the design while improving performance and flexibility.

The end result was much faster, but the look was too simple. It needed just a little something to give the design some depth.

And that is where jQuery came in — I picked up a Drop Shadow plug-in, and put some shadows around the main page elements. It was just enough to make the page look pretty decent.

BUT — performance degraded exponentially every time I worked with web parts. I haven’t determined exactly where or why, but I do know that somehow the jQuery calls combined with what SharePoint does to a web part page in edit mode was very unhappy. As-in, browsers hang and crashes almost every time I worked with a page.

So, I turned off the jQuery when a web part page is in edit mode by just popping the following into the code:

if ($(“.ms-WPAddButton”).length == 0){
…Do the jQuery Stuff…
}

In other words, I use jQuery to look for the ‘Add a Web Part’ buttons. If they are not found, I continue with the script. Editable pages have the simple interface, and run quickly. Normal content pages have the nicer interface… and still run quickly.

There may be a better way to handle this, but it has worked for me so far.

Correcting My Ignorance

A year ago on this blog, I listed some Notes functions that did not have equivalents in SharePoint.

Over this past year, as I learned more, I learned that I was just plain wrong.

So let me take the time to correct my statements,  and lay out the functions in SharePoint to replace the following Notes functions:

• Hide-Whens – Can be computed in your XSLT via SharePoint designer.
• Reader/Author access computed based on document data – Still not available based on my current knowledge.
• Flexible Workflow, sending to different recipients, or selecting a different workflow based on document data – Not available as a single workflow, but multiple workflows can be combined to achieve these types of functions.
• More than 2000 items in a view – Available, but not recommended. Apparently that limitation is not a technical limit, just a performance guideline.
• Private Views – Yeah, these exist. Stored on the server, though, not locally.
• Agents – Not available as a SharePoint function, but you can write scheduled jobs on the server in .NET, etc.
• Action Bar/Shared Actions -Available in Infopath.

There ya go – just reinforces my point that even the most open minded Notes person will not “get” SharePoint development right off the bat. It take times to learn the new moving parts and figure out how to put them together…

“Bad” Technical Answers to Problems

Not much posting recently because nothing profound has been going on. We’ve been moving Notes apps over to SharePoint, doing east stuff first. Simple forms and workflows that can turn into a sharepoint list, calendars, discussions, etc.

Just a couple tidbits we’ve learned:

1) If a user needs to see the old data for a fixed amount of time, we decided migrating the data to SharePoint is wasted effort. Instead, just give them a web interface to the Notes data, either in a new window or an Iframe from the SharePoint UI. Customer are very pleased with this.

2) Icons in columns – We don’t think much of this as developers… sure, we can put up a green checkbox or some other icon in a Notes column. It is such an easy feature, it threw us for a loop when SharePoint couldn’t do something so simple in a column. But it can be done with Dataviews. So we’ve had to pick up some more Designer and XSLT skills to match the options available to us in Notes. Same thing when trying to calculate anything based on “multiple lines of text”. Can’t be done as a standard list option… needs to be done via XSLT in a dataview.

But the real reason I wanted to post was to share our new “Bad” answers to strange, unexplicable problems in SharePoint.

We realized that, as many Notes Devs know, sometimes a problem just isn’t easy to diagnose, and some folks have standarized answers that sound plausible, but which are totally false, and just server to get people off our back while we figure out the real problem. I’ve gotten over this bad habit as I’ve aged, and now freely admit, “I dunno… lemme go figure it out.”, but I still hear plenty of Notes folks who says things like. “Hmm… there must be some database corruption. I’ll go work on it.”. Which often means the same thing as “I dunno.”

So in jest, we came up with the two following statements in cases of unexplained SharePoint problems when you just want the boss off your back so you can go find a real answer:

1) “It sounds like an Office 2007 integration bug.” – for end-user problems.
2) “That sounds like a timer job issue.” – for server-related issues.

Software Engineering Concepts applied to SharePoint

This might be a bit of a stretch, but bear with me…

As programming languages have developed, their evolution has become more like natural language, more intuitive, and well… a lot less like assembly. I feel that systems such as SharePoint are really just a continuation of the abstraction of the code to a high enough level that people can create applications sytems without even writing code at all. These systems are 100% configuration, 0% code.

But… that does not mean that the lessons learned via decades of programming practice across the world should be thrown out. Rather, if we can apply software engineering concepts to the design of our SharePoint systems, we likely will have better systems. Am I wrong?

The specific example that just came to light was the way content types were set up on our intranet. I inherited this SharePoint environment after some high fallutin’ consultants took their checks and ran. But when I saw the way they had designed some specific aspects of this portal, I emailed them basically to ask what the rationale behind their design was. Amongst the actual answers was some advice/instructions:

Create all content types at the root level of the portal. Any new content types that inherit from the root content types must also be in the root of the portal. All changes must be done in the root of the portal.

Now, I get it – they want these to be available to every possible sub-site. But… doesn’t something here sound quite a lot like, “Make sure all of your variables are declared globally.”?

Admittedly, Content Types are not variables. They do not allocate memory just because they exist. Issues of variable scope don’t apply. This is more of a readability/maintainability issue.

It just seems to me that any type of function that is specific to one area of a site could very well reside in only that area. Otherwise, as I must over what the portal might look like in 10 years, I can easily envision a nightmare of content types in the root, with no intuitive explanation of where each is used, what it offers, or why it was created. Is that really what we want?

<petty rant> On another note, I’d like to tell all consultants out there — Content Types are not category fields. If the only thing they do for you is allow you to categorize list items in the UI, don’t create a content type for that. Just make a new column. Call it… I don’t know… maybe… Category? Don’t turn Content Types into your personal hammer just because it is the only feature of SharePoint that you understand. Now run along and spend those big checks we gave you for your brilliant insights into this technology. </petty rant>

The point of all this is really to tryr to express the need to design SharePoint like you would design any application. Not to just treat it as a collection point for hundreds of mini-apps, but to really think out the long-term design of the environment, and create each function acorrding to a broader vision.

Not Working After All

UPDATE:

I found a 64-bit version of the plugin, after much searching. All is now well with the world again, and everything is running.

If anyone else does hit this, search IBM’s web site for WebSphere fixpacks – the DLL that worked for me is found in FixPack 17 for WebSphere 6.1.

Oddly, the broken DLL is in the /bin directory of that fixpack.. I needed to dig down into the other directories to find another copy of the DLL that actually worked.

——————-

I take back my last post. Partly.

SSO works great…. on systems running IIS in 32-bit mode. Like our development environment, where I did my initial testing.

But… our test and prod environments are 64 bit, and the ISAPI filter fails to load. I learned a bit about IIS various operating modes, and based on what I know so far… I cannot leave SharePoint in 64 bit mode, while loading a 32 bit ISAPI filter into a virtual directory on the same web site.

Which means… it doesn’t work.

Now, I could go back to the original plan and put the ISAPI on the Domino servers. But that is not an optimal solution, partly because of the logistics of the controls processes surrounding the various environments. It isn’t impossible… just a very large bureaucratic headache that would take a couple months to get done. Gotta love the corporate world.
It also is not preferred because we would ahve to switch the port on our Domino HTTP process. Which would break our existing links. Putting it on the SP servers lets us update our linked content as we are ready, giving us a very flexible transition on to the new architecture. I don’t want to lose this benefit.

But I find it difficult to believe that we are the first place to try running the WebSphere plugin under 64 bit IIS. Someone else must have hit this problem before. My searches to date haven’t found anything, which means either I am missing something glaringly obvious that would solve the problem, or everyone is sticking with 32 bit. Or there is a 3rd option that I have not thought of.

If anyone has any insights into this nice little twist of technological fate, please enlighten me via the comments.

Changing Plans?

As I’m sure most Notes folks are aware, SharePoint just isn’t as mature of a platform as Domino. While it has quite a bit of potential, and you can bend it to your will, accomplishing almost whatever you want (if you toss in a generous helping of C#), it just doesn’t offer the same day-to-day job satisfaction as doing solid development work in a platform on which you have years of expertise.

As one guy who commented a long time ago guessed, I’m just not sure that working in SP matches my long-term goals. I’ve always thought that I should complete my current migration effort, then switch to a new job, or a new role within my current organization. Some recent events at work (not related to SP or Domino) have made me wonder if I should speed up my efforts and just switch.

Purely from a job satisfaction perspective, SharePoint is drudgery. Learning new techniques amounts to a little bit of time reading up on how they are supposed to work, and a lot of time working around the quirks about how it REALLY works.
I’ve been downright thrilled with my recent Domino work on the other hand — I’ve working with apps that have been around for many years, and have become heaps of mangled spaghetti code. Cleaning them up, re-designing to match the current business needs, and converting them to web interfaces… Truly enjoyable work.

It means I haven’t had much to add to this blog recently, I know. There are plenty of people already covering the Notes/Domino world as-is.

I’m sharing these feelings with all of you not because I think it will help you with your own migrations, nor as a ‘fishing’ post to see if anyone wants a consultant for a while, but just as a statement of how your average Notes guy might feel 6 months into a migration. Morale is definitely an issue to worry about on migrations, as frustration levels run high. I know if I were my own manager right now, I’d be worried a lot more about the people on the team than the status of the project.

IIS and Domino

I’ve been quiet lately, I know. The reason for this is that I’ve been concentrating on refactoring Notes apps recently, preparing them for web enablement as they get integrated with SharePoint.

But one issue we haven’t yet resolved is Single sign-on, authenticating our Notes Apps with Active Directory. I know the theory, and I’ve done it before – run IIS on your server, with the WebSphere plug-in, set some stuff on the Domino side, and everything is working. IT usually takes a little trial and error to get it running the first time, but then all is well.

But here is my question – as part of configuring the WebSphere plugin, you give the hostname and port for your Domino server. Normally, Domino folks are putting the plugin on the same box that runs Domino, but…

Has anyone instead put the plugin on their SharePoint server? Will it send the authentication cross-server to a different Domino box? If so, would that then allow us to use the SharePoint domain name to reference our domino apps, and let the server actually process the correct routing of traffic?

I’m guessing no one has tried this yet – but if it does work like that, it could be a very nice solution – we could have Domino apps thrown into our SharePoint site, and the end users would never know or care what the actual platform running behind the scenes is. This would let us maintain our investment in the Domino platform, while still reaping the benefits of a single SharePoint portal for the user interface.

Evolution of our Mindsets

We have gone through a fairly quick evolution on our opinion of SharePoint since I started this blog. I wanted to break it down, because it seems that people’s attitude towards SharePoint can sometimes flag exactly how well they know the product.

Phase 1: “SharePoint can do anything. Let’s migrate everything.” These are the folks who have read the marketing, but never worked with the product. They just don’t have a realistic sense of what makes it difficult.

Phase 2: “SharePoint sucks. It is a horribly immature product that cannot do much of anything, and any real complexity takes custom .NET coding.” Noobs. People who haven’t yet learned the subtleties of how to piece things together in SharePoint, and what kind of advanceed customizations can be done with SharePoint Designer.  (And yes, I was here when I started this blog.)

Phase 3: “Look at all of these cool WebParts I’ve written to help us get the most from SharePoint” Sadly, most consultants I see are here. They say they are SharePoint experts, but really, they are .NET experts who front-end their code through SharePoint. Their first reaction to anything complex is to fall back into custom coding. But they can do more with SharePoint than most corporate folks, so there are a lot of these folks out there building SharePoint environments.

Phase 4: “SharePoint can do a lot more than people think, if they know how to work it.” This is the first level of expertise that I’d actually hire to work on a project. They understand how to do all the basics, and also can work the XML/XSLT to tweak the UI, they know how to create complex sets of workflows, lists, and libraries, then pull them all together into a slick interface. They know how and when to integrate InfoPath, and how to throw in C# code-behinds instead of giving us custom DLLs and webparts to deploy.

Phase 5: “SharePoint is just a tool. I have expertise with it and can make it do quite a bit, but let’s evaluate it alongside other technology options and pick the best solution.” Here we go. This is where we all want to end up — knowing the strengths and weaknesses of multiple platforms, and matching solutions with business needs.

Of course, no one fits neatly into one of these categories. My team has elements of phases 3, 4, and 5 in their attitudes, but none of us have actually built so much experience that we’d claim experitse in the platform yet.

The reason this really matters to me is mostly for vendor relationships. If you are hiring a consultant, or even an in-house employee, I’d be very wary about anyone whose attitudes aren’t at level 4 or 5. When looking at everything we’ve done in this first year, I’m concerned that we could have done things better, and have built some poor precedents for how we architect apps. We can certainly change the patterns we’ve fallen into, but that actually takes leadership skillz, and hey, we’re tech folk.

Writing Complex Workflows

This isn’t really going to be an in-depth technical post… just a brief description of a change of mindset that our Notes team needed to realize to work with SharePoint:

In Notes/Domino, we are used to a single agent holding all of the logic for a complex workflow. Between LotusScript, Java, Script Libraries, even calling out to system DLLs, an agent can become quite a sizable chunk of code. And we like it that way – it makes the agent list a readable, maintanable catalog of what code runs in each application. New developers can take a quick look at forms and agents and understand how an app works.

So in SharePoint, we were incorrectly expecting to see something similar – a single container that holds all the logic for a complex process. What we are finding is that expectation is making us fail at appreciating what SharePoint can really do. If we accept that a complex process may hold dozens of similar workflows to be called by separate lists, and come together into the full process, we are able to do much more work.

As the perfect example, from the day we sat down in a SharePoint training class, we were in shock that you could not have a workflow branch its logic based on form data. At least, not via the SharePoint designer. What never ever occurred to us is that you branch your logic before the workflow starts. Each branch is its own workflow object, and you call the proper one.

Our mindset needed to change from expecting a single object as the “parent” of all logic, to thinking of each object as its a single function of the greater whole, even though there is no place in the interface to point at as the container for the whole app.