Serious performance problems SBS2003

[Northcountrystan]

I've had several consultants looking at my system and so far we have not found a solution to very serious performance issues.

The system is a Dell 2650 with dual 3 gig processors, dual channel raid controller, 2 gig of memory. DASD is a powervault 220s that is configured with 14 x 73 gig scsi 15,000 rpm drives in a RAID 10 configuration with two hot spares. The backplane is split so we get a total of 6 x 73 useable. The system is running SBS 2003. We are hosting our own websites as well as an Outlook email server. Connection to the Internet is on a dedicated (to this system only) DSL line that has tested at over 750 upload and 3,000 download speeds. Our router is a late model Netopia which replaced a R9100 (and we saw a performance boost). We have a maximum of 30 users on the system, but the average load is well under 10.

When running through our network, the performance is great. Our websites respond in excellent times. However, our VPN connections are driving us nuts.

VPN is used for email, and mapping drives from our server to our users for download purposes. Email response is very acceptable, though not as quick as the hardware configuration would suggest. If I access a mapped drive with the command prompt, I can browse directories within seconds. If I access the same mapped drive with windows explorer the response time is 10 seconds to display the root directory of about 15 folders, and over a minute to look at the contents on one directory with over 2,000 entries. I'm not talking about looking at the contents, just the directory entries. When using FrontPage, the remote response time (our pages reside on a mapped drive) is so slow that we can't maintain our websites remotely. Displaying the contents of a folder (again over 2000 entries) takes so long I could cook dinner waiting for a response.

We have another server that we use to backup our primary system. It is a dual 1 gig and has RAID-5 configured scsi disks similar to the primary. If I initiate a backup from the primary system to the secondary system from the console, the whole backup takes about 15 minutes. If I initiate it remotely, again using tunneling to get to the drives, it takes THREE HOURS. When I watch the system during the backup, the first described backup has the disk drive lights lit like a Christmas tree. In the second situation, the lights blink once in a two second or so interval.

I appreciate any and all help you folks can give, including the elimination of VPN if that is the problem, as long as we can get a high performance solution. Our total needs are: maintain our own websites (12 maximum so far) in both hosting the sites and maintaining the code on the server, host our own email, having the ability to easily and quickly access "mapped" drives or folders on our server to upload or download data. There may be future needs, but this is what we need yesterday.

TIA for any and all help/suggestions, etc.

[cluberti]

Connection to the Internet is on a dedicated (to this system only) DSL line that has tested at over 750 upload and 3,000 download speeds. Our router is a late model Netopia which replaced a R9100 (and we saw a performance boost). We have a maximum of 30 users on the system, but the average load is well under 10.

I'd say the DSL line is likely the culprit - great throughput, but horrible bandwidth compared to a T1 or DS3 connection. Since a DSL line is only two channels, you'll likely see a performance hit when adding more than 3 or 4 users at any one time. The T1 will have 24 channels (well, 23 usable for data transfer), whereas a DSL line (of any speed) will still be two channels. Having a lot of throughput is nice, but the more channels you have on the line (within reason), the better your line will do under load. Even if both connections are rated at 1.5Mbits, the T1 will perform much better than the DSL, always. The number of channels does make a huge difference when the line gets under load.

[Northcountrystan]

Thanks for your response. Unfortunately the horrible remote response times occur when I'm the only one on the system which kind of eliminates in my mind the DSL. Also, the fact that I can vpn in using a msdos prompt and get super fast directory entries displayed, vs 70 seconds using windows explorer vs tens of minutes using FrontPage kind of tells me that the problem is not related to line speeds. I think SBS 2003 is getting lost somewhere either through timeouts... waiting for a response from some internal subsystem or perhaps the problem is related to DNS.

Any other thoughts?

I appreciate your help,

Stan

[cluberti]

Perfmon logs and netmon traces of the server and client when this is happening would be a good place to start. Microsoft has a tool to get you started on perfmon here, called perfwiz:

http://www.microsoft.com/downloads/details...&displaylang=en

Getting netmon installed at both ends and doing a trace on each machine when it is slow will also give you an idea of whether the data across the wire is just slow, or if one end is having issues dealing with the traffic in some way (you'll also be able to see the contents of most packets as well, which can help).

As far as the DOS prompt and accessing file shares, it is very important to note that going through a DOS prompt to access shares and files doesn't require the type of resources and privilege enumeration, both on the client AND the server, that you need to spin up when through the GUI. Using a DOS prompt instead of a GUI to access file shares will be much faster all of the time, not just over slow links. Try it internally sometime - the DOS accesses will always be faster than using Explorer or an application's open or save dialog box (which also uses Explorer.exe, and thus shdocvw.dll, shell32.dll, etc).

I'd still say you could have latency issues on your DSL connection that you wouldn't see otherwise - again, only having two channels is still possibly an issue (you've told me nothing that would rule this out yet). Remember that a VPN connection also adds TCP overhead to each connection and each packet sent across the wire.

Edit: I need a spellchecker.

Edited December 9, 2005 by cluberti

[Northcountrystan]

Thanks. I'll pass this on to my consultant.

Stan

[cluberti]

Good luck. Just to shed some more light on my opinions of the issue from what you've told me, I've seen this many times before:

1. There may be some misconfiguration on the client or server end (gateway, router issue, bad cable, etc.)

2. There may be a performance issue with the server itself (the issue should happen to all users, all the time in this scenario, and it isn't now - but I'd still perfmon the box for a few days to see if it's behaving within "normal" constraints, memory/CPU/Disk/process-wise.

3. The internet connection is more than 70% saturated, more than 50% of the time, in which case it's time for a line upgrade.

4. The internet connection has a high latency caused by poor wiring, a routing issue within the ISP's network, a deliberate DDOS attack against the circuit, or just the victim of a long run from the demarc to the ISP's upstream equipment.

I've consulted for many, many years, both as a network and a platforms consultant. I also spent many years as a network engineer, so I do speak with at least a modicum of knowledge about the subject. One constant I find with businesses with issues such as these is that they either aren't using a full T1 when they should be, or they're trying to save some money with a DSL or cable connection. Most people don't understand that the performance you see on a network link is affected by both throughput AND bandwidth, and even fewer people understand the difference between the two (they were seen as one and the same by quite a high number of people I met as a consultant). Even an upgrade to a fractional T1 (768K or higher) will result in better performance than a 1.5MB SDSL or cable link for business use patterns such as this - it's not the throughput that affects performance here, so much as the bandwidth available.

Edited December 11, 2005 by cluberti

[Northcountrystan]

Thanks for the input. I'll download the microsoft tools you suggested and install them. As to fractional T-1 lines, that is out of the question as I'm in a rural area and would have to bring in the connection from over 80 miles away. My only choices are roadrunner (which we are replacing) and DSL. Our speeds have been tested at over 750/3000. What is troubling to me is that I have stellar performance on my websites yet I can't use Front Page because its performance is zero. The performance on roadrunner is the same and have paid to have 3/4 meg up and 3 meg down which tested out fine. The reason we are moving to DSL is to maintain 24x7 100% uptime (as close as possible). My office is on UPS and generator backup, but when power is out in the area, cable is out. DSL stays up.

From a hardware perspective, I can't believe there is an issue. My cpu utilization is 1 percent (not a typo) and my DASD performance is almost a terabyte per second (raid-10 with two sides of 6 striped 160meg / 15000 rpm drives). I think it would be very hard for me to find a faster hardware platform then I have.

I hope that the monitoring tools you recommend will shed light on the problem. BTW, although this is in a real estate office, my background is over 37 years of data processing in all aspects including owning a consulting company. My problem is that I have 37 years in IBM mainframe computing, and this Windows stuff drives me nuts. To me the software is illogical and clugey, but it what I have and need to make it work.

Thanks for your input and I'll be trying out the software you recommended. If you have any other ideas, please let me know.

[cluberti]

I think you'll have good luck with the network monitoring tools, and I too believe that your hardware is not the issue (you've actually convinced me it's your line by stating that web pages load fine but FrontPage takes forever). Again, good luck, and let us know how it goes.

[Ghostrider]

I know this is a little simplistic an answer but having tried to work within the confines of customers budgets and trying to use VPN across DSL connections i can now categoricaly say... YOU SHOULDN'T DO IT... it sucks... Bell Canada have also recently (quietly) admitted that there are serious issues with VPN across DSL and on lesser routers your connection will drop, you have a good router so it keeps the connection but i believe througput will be very bad. as mentioned earlier i would try other alternate options, even 802 wireless, 1 of my clients is working this way when DSL failed...

[Gekko_uk]

Just for your info.

I have a near Identical Setup as you..... and I have the exact same problem.

I would not let the DOS Via GUI compraision confuse you, ie if you look at your C drive in Windows and then in MSDOS you shall notice that DOS is quicker here - for the same reasons as someone else mentioned.

I just put this problem down to my connection....... but as you said, you have a connection similar to mines.. good luck mate

Cheers

[Northcountrystan]

Folks,

Thanks for the input. We are working on this issue this week. Frankly, looking at 1% cpu and disk storage that is about as fast as you can buy, with an underutilized network, I'm ready to move on to something other then Microsoft. I run an IBM Mainframe on a single DSL line and have great response time. My websites run fine on this particular DSL line.

Linux is looking much better to me lately.

I appreciate the help and will report back when and if I have resolution.

My bottom line: Microsoft software sucks.

I know this is a little simplistic an answer but having tried to work within the confines of customers budgets and trying to use VPN across DSL connections i can now categoricaly say... YOU SHOULDN'T DO IT... it sucks... Bell Canada have also recently (quietly) admitted that there are serious issues with VPN across DSL and on lesser routers your connection will drop, you have a good router so it keeps the connection but i believe througput will be very bad. as mentioned earlier i would try other alternate options, even 802 wireless, 1 of my clients is working this way when DSL failed...

[cluberti]

Microsoft software sucks.

Your opinion. Personally I think Microsoft products are just fine, but use the best tool for the task at hand - if Linux or Unix are better tools for you, I suggest you use those tools. Again, best of luck to you.

[Ghostrider]

Microsoft software sucks.

Your opinion. Personally I think Microsoft products are just fine, but use the best tool for the task at hand - if Linux or Unix are better tools for you, I suggest you use those tools. Again, best of luck to you.

I agree that each to their own.... I do actually like Server 2003 and even better the SBS version and find they work very well, However, my problem has always been the internet connection. DSL is not made for VPN, you need T1 or Fibre, a good Wireless connection is better than DSL, (even cable works better ) We would all be interested to know what the actual cause turns out to be... best of luck in finding it...