Road map for functions in the SharePoint World

When people say “SharePoint”, they usually mean any application accessed through a SharePoint portal. In truth, this may actually be SharePoint, but could also mean .NET or InfoPath applications.

One of the challenges we’ve had is defining which types of applications really are a SharePoint app vs. InfoPath vs. .NET.

But before I go there… why does it matter?

It matters in our organization because of the various roles certain individuals play. Power users and business analysts can do some of their own work… if it is truly pure SharePoint. We’re finding that Domino developers pick up InfoPath so well, they can do more with it than many Microsoft experts. (Which is a blog topic in and of itself, but in short, we’ve spent years working with form-based platforms… we know what to do with, for example, Conditional Formatting (Hide-Whens) to simplify forms instead of making new forms and views for minor UI differences.) And… .NET — while our team can certainly write the code, it brings on whole new levels of deployment and maintenance issues. So we try to avoid it.

Which brings us to the “Road Map”. Which is not very profound… it is a very simple thought exercise for any application function that we are asked to provide:

1) Can SharePoint do it? Then make it so.

2) Well, then, can InfoPath do it? Then make it so.

3) Well, then, we’ll have to write .NET code.

After some discussions, we also added a question 0:

Does this even need server-side code? If JavaScript in the browser can do it, don’t get too clever with SharePoint… just write the script.

We had to add this question because developers who have never really been web-focused are forgetting that SharePoint ultimately renders in a web browser. Throwing some script in a master page or even in a content editor web part is sometimes a very quick, easy solution to some problems.

Our most recent example of this was a ‘Contact Us’ link. It was supposed to be available on all pages, provide an infopath form, then return the user to their original page on submit. They first wrote .NET code to do this. I talked them into just using a SharePoint list. Then they were doing some crazy SharePoint CAML stuff to make it happen. I talked them into just using JavaScript to append ‘&Source=originalpageURL’ to the HTML link.

I use this as an example to illustrate the eternal point of KISS — if 1 line of JavaScript code does the trick, don’t go writing .NET features. Find the simplest solution.

The fun part of migration efforts

There is one part of migrating to a new platform that I am truly enjoying — the cleanup work on the Notes/Domino system. I love taking an old, cumbersome app, and refactoring the code into a quick, simple design with significant performance and UI improvements. I can usually reduce the support needs at the same time, frequently reducing the headcount required to support the apps by well over 50%.

We end up doing this kind of refactoring on many apps before thinking about a migration because many of the apps have have devolved into such massive heaps of spaghetti code tht they need a good refactoring just to understand exactly what the current functions are.  Once the refactoring is done, then we talk to the business about whether all components are still needed, and look at whether or not a change in a business process may allow us to simplify the app even more. What we find is that through this simplification process, the migration becomes only partially technical, but as often as not, a business transformation is the result of the technical work. A a chunk of the time, the resulting app is so easy to use and maintain that we just stick with it, and don’t migrate it after all.

Which leads me into my next train of thought – if the refactoring is what I enjoy, and what I excel at, why do I spend so much of my time on the areas that I do not?

So I am considering taking on side projects from other organizations. Seeking out other people, whether or not they are migrating, would would have an interest in hiring me to refactor their “ugly code”. For now, I am thinking just 5-10 hours a week on the side, unless someone has a project so major that it makes it worth switching to an independent consultant on a full-time basis.

So what do you all think? Is that something you would have any interested in contracting out?

Disconnecting the Data Infrastructure

One of the challenges we’ve encountered as we migrate application off of Lotus Notes and over to SharePoint is what I call the “data infrastructure.” There may be a better term for this, I dunno… but I am talking about those Notes databases that are shared data repositories for the enterprise, but not applications unto themselves.

For example, we have a database with a record for each employee, with their Job Title, Location, Phone, etc. We have another one for just IT employees that tracks who falls into specific support roles for each office location. And yet another that holds generic keyword lists… our office locations, department lists, etc, etc.

Some of this may be stored in the NAB in some organizations, too… but that doesn’t change the problem — one way or another, you need to replace these data sources with new ones.

For some (most) data, just a new SharePoint list is sufficient. Or multiple lists. But the hassle comes in data conversion, and/or data maintenance, especially if there is yet a deeper root data source that was populating the Notes data.

What I have done to make this process go forward is the following:

1. Create the new data sources first, before even trying to migrate any apps.
2. Create Web Services to publish data from the new source as XML, thereby making it available to either Notes or SharePoint (or anything else for that matter).
3. Disconnect the Notes apps without migrating – go into the code and just update it to read the Web Service, without making any other changes.
4. Once everything is disconnected, kill off the old data databases. Now you can go forth and concentrate on migration efforts.

One of the key pieces of information that got me going down this path was the (not-so-profound) epiphany that AJAX calls are not limited to web browsers. They work great in the Notes Client, too. The XMLHTTP object is just an object in the OS like anything else:

Set xmlhttp = CreateObject(“Msxml2.XMLHTTP”)

So I just declare the object, call my Web Service’s URL, and parse the resulting XML for whatever purpose is appropriate. Once the data is available in LotusScript, any existing code can take over from there.

I haven’t thought of any efficient way to make this work in @functions… I suppose you could use @URLOpen to pull in the Web Service data, but I suspect by the time you got done parsing everything, it would have been easier to just re-write your code in LotusScript.

Software Engineering Concepts applied to SharePoint

This might be a bit of a stretch, but bear with me…

As programming languages have developed, their evolution has become more like natural language, more intuitive, and well… a lot less like assembly. I feel that systems such as SharePoint are really just a continuation of the abstraction of the code to a high enough level that people can create applications sytems without even writing code at all. These systems are 100% configuration, 0% code.

But… that does not mean that the lessons learned via decades of programming practice across the world should be thrown out. Rather, if we can apply software engineering concepts to the design of our SharePoint systems, we likely will have better systems. Am I wrong?

The specific example that just came to light was the way content types were set up on our intranet. I inherited this SharePoint environment after some high fallutin’ consultants took their checks and ran. But when I saw the way they had designed some specific aspects of this portal, I emailed them basically to ask what the rationale behind their design was. Amongst the actual answers was some advice/instructions:

Create all content types at the root level of the portal. Any new content types that inherit from the root content types must also be in the root of the portal. All changes must be done in the root of the portal.

Now, I get it – they want these to be available to every possible sub-site. But… doesn’t something here sound quite a lot like, “Make sure all of your variables are declared globally.”?

Admittedly, Content Types are not variables. They do not allocate memory just because they exist. Issues of variable scope don’t apply. This is more of a readability/maintainability issue.

It just seems to me that any type of function that is specific to one area of a site could very well reside in only that area. Otherwise, as I must over what the portal might look like in 10 years, I can easily envision a nightmare of content types in the root, with no intuitive explanation of where each is used, what it offers, or why it was created. Is that really what we want?

<petty rant> On another note, I’d like to tell all consultants out there — Content Types are not category fields. If the only thing they do for you is allow you to categorize list items in the UI, don’t create a content type for that. Just make a new column. Call it… I don’t know… maybe… Category? Don’t turn Content Types into your personal hammer just because it is the only feature of SharePoint that you understand. Now run along and spend those big checks we gave you for your brilliant insights into this technology. </petty rant>

The point of all this is really to tryr to express the need to design SharePoint like you would design any application. Not to just treat it as a collection point for hundreds of mini-apps, but to really think out the long-term design of the environment, and create each function acorrding to a broader vision.

LOE comparison – Notes vs. SharePoint

For those keeping track, our formal strategy on what to do with Notes vs. SharePoint was on hold pending some hard data on the level of effort to develop and maintain applications.

So we did what any group of people without data would do — we made some ourselves. We selected an app that was basic enough to be realistic, but with enough quirky features to throw a wrench at us, and went ahead and made a prototype.

Results were as follows:

• 1 week to get the development environment all worked out, mostly by finding the right patches for Office and SharePoint.
• 4 days of learning the various controls and techniques available to us.
• 1/2 day of actual development once we got it all put together.
• Approximately 80% of functionality with an exact match, 15% with a reasonable change of function to met the same business needs, and 5% not achievable with a realistic amount of effort.

And we concluded the following:

• For strategic purposes, we should ignore the 9 days of “learning curve”. That is a one-time effort for each developer, probably decreasing as the team’s skills grow.
• The actual development effort between SharePoint and Notes is fairly comparable, IF you stick with certain restrictions (listed below). Details of specific apps may make one side quicker than the other, but the delta is small enough that it shouldn’t impact our strategy.
• Security is weak in SharePoint. Before SharePoint folks jump on this, let me give a caveat to that statement – You can make SharePoint security match every Notes security function short of field-level encryption, BUT that then changes the level of effort significantly. To keep development levels on pace with each other, apps that require complex, strict security need to remain within Notes.
• You must use InfoPath. We, as developers had decided that InfoPath was nice, but not really needed. But the time savings in development has changed our minds. Without InfoPath, stick with Notes.

The bottom line is this — from a purely technical point of view, migrating to SharePoint is a “neutral” activity. It neither offers great benefit, nor does it offer great harm. The decision is going to be made on a non-technical basis. The cost of running SharePoint is likely higher today, but it will likely come in line over the next couple years as skill sets improve and Microsoft improves the product.
Technical folks will be able to punch holes in SharePoint to their heart’s content… there are enough gaps and issues to do so. However, SharePoint will meet a significant majority of requirements, and the amount of technical effort required to fill in the gaps will be an acceptable risk/cost to most organizations with valid strategic reasons to be moving to SharePoint. In particular, if a small/medium sized Notes environment is kept around for the ~10% of functions that SharePoint cannot handled.

Our overall recommendation in our organization is to do new development in SharePoint, web-enable as much Notes as we can, remove the Notes client from the organizations, and let the Notes system decrease over time through natural attrition, as the SharePoint environment matures.

(Note that this is a change. When I started this blog, the strategy was “Get rid of Notes. Now.”)

This path also leaves us with options – if the platforms undergo major change for better or worse, we can always turn our strategy around.

Synthesis of Discussions on Notes v. SharePoint

In the past week, I’ve received numerous emails, comments, and reactions to the general concept of migrating Notes to SharePoint.
I wanted to post a brief synthesis of some of the main points, and add some commentary of my own:

1) Notes vs. SharePoint is not comparing apples to apples. Quickr is the better comparison tool.
This is true. However, Quickr does not have the market penetration that Notes in general does. Much as I’d like to say it is a fair discussion for organizations in my situation, unless you already have Quickr going in your environment, it just isn’t realistic. We’re talking about determining the best platform for our current functions, not evaluating a new platform for future collaboration. Quickr just is not a consideration in our current environment, for political, if not technical reasons.

2) Web enabling an entire Notes environment is a difficult, expensive task.
Maybe it is because I’ve been web-enabling Notes apps since the first IBM-internal beta of R5, but I just don’t think it is as difficult as people are making it out to be. There is certainly a large learning curve, but I got over that curve almost 10 years ago. Am I the exception? From all the wonderful tips and techniques offered in the blog-o-sphere, I thought more Notes developers were comfortable in this arena. Is my perception inaccurate?

That being said, I agree that a 100% web enablement would be a chore. We’re more likely to pick the easy apps, and pick the apps with a large user base. If we can web enable enough apps to decrease the number of Notes Clients in the environment, we decrease support and maintenance costs. The more we decrease those costs, the more management will support keeping Notes/Domino as a technology platform.

3) Why would you migrate anyway?
It all comes down to cost. I think everyone will agree that today, Notes is cheaper to run. We already have a talented staff, the environment is stable, we have processes in place, etc. In addition, truly talented SharePoint experts are rare and expensive. A lot of people have it on their resume, but they crash and burn in a real project situation.

So with all that in mind, how can SharePoint save money? People I have spoken with believe that in 3-5 years, the costs will reverse. 3-5 years from now, SharePoint folks will be as common as C# folks are today. Microsoft will have released one or two more major releases, and will respond to the major weaknesses being identified in the platform. (With Ray Ozzie at its helm, no less…)

On the other hand, Lotus folks are a dwindling population, at least in my area. I’ve talked to a number of organizations who will accept a greater cost today to get to the platform that they believe will become cheaper as Microsoft responds to the weaknesses of SharePoint, and the talent pool increases their skillsets.

After all the talks, my general strategy hasn’t changed much from one of my previous posts, and it is very close to becoming the official strategy at my workplace:

1.  Increase the internal skillset on SharePoint, BizTalk, .NET, etc. to reduce the amount of consulting required on the platform.
2.  Web enable Notes apps, and integrate them with a SharePoint portal, striving for a reduction in Notes client deployments.
3.  Create new apps in .NET/SharePoint, unless Notes has a clear cost and speed advantage for the specific functions requested.

Let me just add one or two caveats:

• If you are already on Quickr, your strategy may be different.
• If you are so large that the enterprise-level weaknesses of SharePoint are unacceptable to you, Notes makes more sense.

And I would like to close with an open-ended question — given the following root causes of the desire of “management” to migrate to Notes…

1) Decreasing Lotus talent pool, and increasing cost of that talent.
2) Confidence that MS will resolve their SharePoint issues, and decrease long-term costs and pains in a SharePoint world.

….What can IBM/Lotus do to turn around migrations like mine?

(From a truly unbiased point of view, I shouldn’t care. But with 15 years of Notes experience, but only 2 years of .NET, and 6 months of SharePoint experience, I just plain got more skillz on the Lotus side, and I’d like to use them.)

Soliciting Feedback

I’m looking for some answers from the community…

As I mentioned previously, the status of Lotus Notes/Domino is again under discussion. To make an educated, unbiased business decision, we are looking for data on the following questions:

1) Development Effort – In General, how does development time in SharePoint compare to development time in Domino. Assuming skilled talent in both technologies, and non-trivial apps…  does anyone have any real numbers and data comparing the level of effort to write applications on both platforms? Or even any anecdotal data?

2) Maintenance Effort – How does the level of effort compare to maintain SharePoint Applications vs. Notes Applications. (Again, in general). I’m not talking about the infrastructural support for the platform – just the application support. How many developers does it take to support a 500-app Notes platform vs. a 500-app SharePoint platform?

If anyone has any real world experience, and in particular measurable data dressing these issues, please either comment here or contact me directly at migratenotes@gmail.com.

Thanks!

Admin Vs. Dev

A few months back, I had stated that setting up a basic SharePoint environment really isn’t all that painful. And that is still true.

However –  what I have found is that only basic dev or proof-of-concept environments actually stay that basic. We have about 2 dozen servers in our environment, for example.

And we have spent the VAST majority of our time on administration issues, not on development efforts. In theory, everything will be wonderful once we get everything stabilized. But getting there is taking more effort than anticipated.  Every time we work on one issue, we find three more. The theory sounds much better than the practice.

I think that SP will still come through for us. But the process to go from nothing to a full-blown stable multi-server SharePoint install that actually benefits a business organization… Whew.  It is a doozy.
There was always a half-true joke in the consulting world that to accurately scope out a project, you take the estimate from the technical team and double it. For SharePoint, I’d double it again. If you are hiring consultants to do it, double it once more for good measure.

SharePoint is not a Sandbox

It is too bad that I have a policy against writing anything directly about my projects at work.  Because it stops me from getting into some details that would likely interest this audience. I can say this much — I have been reading for a long time that running a Microsoft platform will cost more than a Notes/Domino platform. After the past week, I now understand why that is. Readers in the past have question how long it took to get a SharePoint environment up and running. That is still the quick and easy part. The difficulty comes not in the setup… but in the maintenance.

Maybe I just haven’t figured out the best practices yet, but it seems like SharePoint is designed to require ongoing maintenance. The theory is that the end users can manage their own sites, and IT just need to keep the environment running and do special projects. But wait — this sounds exactly like the Notes world back in its early days. At that time, IT shops thought Notes could just be thrown to the users, and they can create their own DBs without much help from IT. And some shops did do that. But most of the places I worked did not… instead, the IT shop handled every single request for a new Notes app, they handled every maintenance issue, and eventually the Notes environments were locked down fairly tightly, with end users having freedom only within specific areas of applications pertinent to their business.

I would be surprised if SharePoint did not go that same route. Business users will probably really like being able to throw out a new document library, discussions, etc. But I don’t think they will like creating lists. I doubt they will want to bother with the tedium of creating columns and views, and I doubt they will enjoy learning about all the different data types and function you can do in a list.

If I had to predict the future, I’d predict that SharePoint environments will degenerate into a mishmash of document libraries, with a high-level structure generateed by IT, and utter chaos once you drill down to the areas controlled outside of IT.  In a couple years, the major SharePoint projects will not be setup projects, but cleanup projects. We will spend our time reorganizing content, locking down access, and treating SharePoint like one large application (which it is.)

The idea of collaboration gets confused with the idea of complete freedom within an application. Collaboration works best when there are enough controls in place that everyone works in the same way. SharePoint does have the power to create a good collaboration environment. But it needs IT oversight. Repeat after me – SharePoint is not a Sandbox.

Now this is an app that should migrate quickly…

I normally refrain from talking directly about work, but this app deserves special mention because it is a rare one – it practically screams, “Migrate Me!”.

It gathers data from the end users, then lets them select their own approvers, and when approved, uses a Word Template to copy their data into a printable Word Document.

The way it is done in Notes is very fragile. It uses OLE Links to the Word Templates. It creates multiple NotesDocument objects with the updated documents, and again uses an OLE Link. It then activates that link to display it to the user. Of course, if your files ever move, your app is broken. I can think of better ways to achieve the same result in Notes…

But why even bother? If your requirements are for Word documents with a specific format, SharePoint becomes a clear and simple answer.