As the developers had been using VB/ASP for some years, we immediately suspected that long string handling was the most likely cause of the performance problems. Some tests quickly proved this. In VB/ASP as a string gets longer and longer, the performance of operations on the string degrades faster and faster (possibly even exponentially). Our application was running on IIS and using ASP, so we could have written a VB dll to handle the strings, but as the string handling problem also affects VB, a better coding methodology was considered appropriate.
At this stage some readers, of this article, will be suggesting that we should have written a dll using C++ (or another 'more efficient language'), but our skills in other languages were not up to that task. We needed a solution to run on a web server, in a language we could use, and we needed it quickly (as usual).
We tried a few different solutions, but in the end settled on splitting the long string into an array of smaller substrings. We could then pass through the array performing operations on much smaller strings, before recombining them into a long string again.
The results surprised us.