Pipe broke..

[Porn Loader]

so i got a call from my boss saturday morning. apparently a pipe in our heating loop broke. where was it? right above my racks. nice friggin design. i want to shoot whoever put the room there without looking at the HVAC schematics. going to be replacing about $30k in gear as it all got soaked. any how first things first company background:

smaller company. about 100 employee's overall, with 5 physical locations (one was a small local office closing in april) across the usa. sharepoint portal and email hosted off site so its basically just AD and file/print at the offices. branches use cisco pix 506e's for firewalls/tunnels, and have their own dc. ok you guys will get a kick out of this as it is the most complex "small" office you'll see

main office:

old compaq brick: (running a pair of pentium pros ) backup gc running 2k3, dead thank god

compaq dl-360 1u: 2k3, fismo role holder for the company. getting replaced but running (dumped about 20 oz's of water out of it)

dell poweredge 1850 1u: 2k3, file and print. wet getting replaced

sonicwall pro 2040: wet, getting replaced, voltages are out of wack on the built in monitor but running

cisco 3005 vpn concentrator: wet but running replacement optional

3x cisco catalyst 3550 POE SMI switches: all dead, pieced one together out of parts. had a power transformer out of a previous one that died. out of service now; got a replacement

cisco 2620 edge router: of main office; wet still running replacement optional

cisco 1751 router: handled a transport T1 to one local office, dead no replacement needed more than likely as that office is closing, killing the transport

d-link layer 3 switch for dmz: wet getting replaced, running now

dell dlt tape backup: soaked, getting replaced

4x 48 patch panels: soaked, getting replaced

T1 for internet: operational

PRI: operational

i think thats all the gear, but ive put in like 50 hours since saturday morning so i may be forgetting something.

internet->2620 (public on both interfaces)->dmz switch->sonicwall (one public, one private int)-> internal

..........................................................................from dmz^->3005 vpn concentrator (one public, one private int)-> internal

anyhow heres my biggest issue. we run a cisco callmanager 7750 for phones at the home branch, and also the closing office across the transport. it was dry, except for the top. all modules were dry, no water in the bottom, all modules dry. but im still worried about it just based on cost. havent looked recently, but most of the cisco voip is going for $30k plus. worried about corrosion/oxidation down the line

heres my plan. all the branches are up and running except for the local "closing" office (stole some gear from there just in case it was needed).

get $$ for the concentrator, no replacement. the replacement sonicwall can terminate the tunnels just fine. christ its just three tunnels.

get cash for the cisco 2620, i've got a 2650 at the closing ofiice i can use for an edge router

no replacement for the 1751 as that office is closing

now we are getting to the interesting part. the vpn conc was parallel to the firewall (standard cisco setup), but leaves it as a possible breaching point. my default gateway (server was pointing at the switch) was one of the 3550's (all were connected by gbic). now on the default gateway had ip routes pointing at the concentrator for remote offices, and the transport T1. open up the sonicwall and it had about 6 ip routes. where where they pointing? at the default gateway aka my main 3550. talk about a loop... reset it factory defaults, as anything that makes that far it should go out the firewall.

basically everything that got wet is gone, except for the callmanager. only going to order 2 new servers (probably a pair of the 1850's unless dell actually starts selling the opty's now), but my main question is does anyone see anything out of wack with my plan? any opinions?

as for switches i already picked up a cisco 3560 48 port POE SMi switch, and will more than likely pick up another.

Edited February 22, 2006 by Porn Loader

[RogueSpear]

WTF.. you got about 5x as much equipment as I'm using for 200 employees and I don't feel constricted in anyway when it comes to the networking hardware (servers are another issue).

[LLXX]

Pure water? Everything should be fine once it dries out completely. Pure water doesn't leave behind any residue when it evaporates.

[Porn Loader]

Pure water? Everything should be fine once it dries out completely. Pure water doesn't leave behind any residue when it evaporates.

nope, just tap water. and as a side note.. when i got here everything still had power and most everything was still running. pulled everything out and started drying it off w/ paper towels and a heatgun.

rogue, the best part is on average there are only about 11 employees in this office it varies though. we are starting up a toshiba rdm project that i'll have to set up a call center here for. that'll double the number of employees in this office

Edited February 22, 2006 by Porn Loader