Downtime in the Desert

Six hours after the e-mail system went down, Derek Kruger fielded his 39th anxious caller. "When will my e-mail be up again?" As IT and communication supervisor for the city of Safford, Ariz., Kruger is responsible for the networking and telephony systems for all city government departments. The Microsoft Exchange

Safford, a town of nearly 8,950 residents located 130 miles northeast of Tucson, serves as the Graham County seat of government. Its city government is spread among 10 buildings, including City Hall, fire and police departments, library, facilities yard, landfill, golf course, city attorneys and the Boys & Girls Club of Safford. Kruger and his team of two engineers and a help-desk administrator support 250 network and telephony users, radios for public utilities and public safety, as well as city-issued cell phones for a majority of the city staff.

The city government was supported by a metro area network (MAN) comprised of 180 PCs, five Dell servers, several disk-based RAID systems and an aging tape system. Kruger and his team had worked over the past three years to connect the city buildings, linking eight buildings with fiber-optic cable and one with a wireless connection.

Continuous availability, data protection and backup presented a challenge to Kruger and his team. Network users saved their files to a server, which was backed up to tape. "By nature, the employees we serve generate a lot of files, many of which need to be held onto for a fairly long period of time for legal or compliance purposes," he explains.

Additionally, the city's servers included a Microsoft Exchange e-mail system and SQL Servers, as well as financial and accounting software. The tape system could no longer keep up with the volume of data to be backed up. The IT team had to back up the Exchange server one night and the file servers over the next two nights. This practice left gaps in data protection and took all night to complete.

The tape systems' limited capacity only allowed for archival of two days' of data, thus inhibiting the networking team's ability to retrieve historical data. "Our limited capacity put us at risk," says Kruger. "If one of the city employees needed to retrieve old documents or e-mails, we might not have been able to obtain them."

TIME FOR A FAILURE

Files or e-mails that could be retrieved were done so in a manual, costly and time-consuming process. Even with these efforts, many times the requested mail or files were unrecoverable.

"Much of our equipment was due for replacement," says Kruger. "Despite our implementing various methods to keep the network up and running and the data protected, we knew it was only a matter of time before we had a major failure."

That day came during the summer of 2007. "We called it the perfect storm," says Kruger. "The e-mail system had been down for almost a week. We had spent a full day restoring the Exchange data from tape. A software upgrade had left a SQL database corrupted, requiring the laborious process of rebuilding it from the transaction logs. We discovered file system corruption on a file server-corruption that was being backed up-that took a couple of days to fix."

Kruger, who had been expanding the city's network since he started working there in 2005, began looking for a solution for these problems. Some of the ailing servers would be replaced. "Obviously, a larger tape system was not going to meet our needs," he offers. "The restore process was just too lengthy and difficult. We needed to recover as quickly as possible and stop these outages to our Exchange and SQL servers."

Kruger knew he would need to propose new equipment to his boss in the city's budget office, as well as present the plan to the Safford City Council. He and the team set standards for the network, recovery point objectives and recovery time objectives that would ensure a greater rate of uptime and data protection.

"As we sought proposals from solutions vendors, we set a requirement of restore in no more than four hours," he says. "We needed a continuous availability solution with near zero data loss, but also needed to keep within our budget."

RESTORE SOLUTION SELECTED

Kruger researched a number of business continuity and continuous availability solutions, and ran across Asempra's Business Continuity Server (BCS) solution. "Asempra's BCS seemed to meet my requirements, so I went for a demo of it at a regional trade show in Phoenix," he recalls. "The demo consisted of deleting a movie that had been playing. They were able to start the movie again in seconds, while the restore ran in the background. Within minutes, the movie was fully restored."

Kruger also liked the solution's ability to protect SQL, Exchange and Windows File Server applications and data through an approach that captures and indexes all data events, creating a virtualized data store that allows instantaneous recovery to any point in time. "The virtualized recovery and data protection technology made it possible to recover Exchange and SQL in 30 seconds, dramatically lower than the four-hour requirement I had set for solutions under consideration," he says.

Kruger worked with Asempra to identify the service-level agreements (SLAs) that would meet the goals and objectives of the various city departments. Asempra designed a solution for Kruger and his team that utilized a single BCS appliance to protect one terabyte of data on two SQL servers, the Exchange server and several Windows file servers.

With this configuration, the BCS exceeds the city's Exchange and SQL recovery SLA, now measured in seconds. Continuous data protection technology ensures there are no gaps in data protection. At the same time, the BCS lowered the costs of availability, data protection and recovery.

"It's great to be able to serve our customers faster and more effectively," Kruger says. "Now, when someone calls to ask for a file that they accidentally deleted or lost, I can just roll back in time to the point before the loss occurred."

Since the deployment, the city now provides true continuous availability for its Windows environment, is ensured against data loss and has substantially cut its costs compared to the previous tape infrastructure.

The Asempra BCS, connected to the city's LAN, leads to a newly implemented storage area network (SAN) connected to two MPC storage arrays, located in different buildings. Each array has 5 terabytes of capacity and both are configured as a mirrored pair.

"The SAN and mirrored configuration gives the city a great level of redundancy," says Kruger. "If one array goes down or was to be destroyed, the other immediately takes over the storage functions while protecting the stored data."