In the spirit of storage. (Cover Story).

Southern Wine & Spirits of America is one of the largest

wine and spirits distributors in the country, with more than $4 billion in revenue and commanding 12% of the total domestic wine and spirits revenue. Operating in 10 states, it represents more than 300 suppliers and 5,000 brands, and services 125,000 retail and restaurant customers. Its 6,000 employees operate a distribution network that stretches from Miami to Maui.

While the company has gained a dominant position in its industry through its top-of-the-line warehousing and distribution network, its storage network was another matter entirely. Due to rapid organizational expansion, coupled with the explosive growth of storage consumption, server sprawl became the order of the day.

Networking personnel spent hours each day dealing with storage issues and backups became almost impossible to complete. This situation became most apparent with subsidiary Southern Wine & Spirits of California (SWS), the state where the company enjoyed its largest market share (more than 50%).

"Backup times were killing us, we were suffering serious amounts of downtime and having to add two or three servers each month just to keep up with storage demands," says Robert Madewell, director of networks at SWS.

SWS employs 2,000 people in California and maintains a distribution network comprised of executive, warehouse, transportation and training facilities. This network is managed from two sites--Union City in Northern California and Cerritos in Southern California.

TRAFFIC THROUGH TWO LOCATIONS

A total of 70 HP Proliant servers are installed between both locations, with traffic divided functionally rather than regionally (i.e., certain applications are run on servers at Union City and others are housed in Cerritos). Thus, depending on the application required-primarily Exchange Server for e-mail, proprietary collection software, a sales reporting system, SQL server databases, fax servers, Web servers and call center servers-network traffic was distributed accordingly using primarily Cisco networking gear.

Most servers ran Windows 2000, though some NT units remain to be upgraded. While the network itself could cope with the load, lack of adequate storage management often ground things to a halt.

"We were experiencing severe storage issues at both locations," says Madewell. "With storage consumption growing at a ferocious pace, coupled with no way to track usage, we would be faced with losing servers a couple of times a month due to them filling up."

Once full, users would be left with nowhere to store newly created documents. That led to lots of calls to IT, hours wasted manually correcting the problem and system downtime. According to Madewell, IT staff spent an average of two to three hours each day putting out storage-related fires and troubleshooting network storage issues--a big distraction for the eight staffers who had seen storage consumption grow fivefold without any more personnel being added to the unit.

In one year, for example, storage usage grew from 170 gigabytes to 400 gigabytes in the north and from 90 gigabytes to 200 gigabytes in the south. As well as lost time for users and networking staff, this growth also led to the addition of two or three servers each month just for new storage. Madewell also notes the addition of server racks, more power to server rooms, a larger AC unit and three more Cisco switches in the same year.

"We would spend hours pouring through volumes on the affected servers to see what space could be freed up," he says. "In many cases, they'd find large collections of PowerPoint presentations, MP3s, JPEGs, aging personal files, and multiple copies of the same documents residing in directories, as well as e-mail databases."

SMART FILTERING IMPORTANT

With no way to track storage usage centrally, two IT staff would go through the various directories, checking files one by one to determine which were of value and which could be deleted. One group of MP3s, for example, was almost deleted, until IT discovered the marketing department used the files frequently.

That near misstep demonstrated the need for smart filtering of stored files. "We would have been in big trouble if we had just eliminated those files," says Madewell. Meanwhile, SWS struggled to cope with backup demands.

A full backup using Veritas Backup Exec took from Friday at 6 p.m. until late on Sunday. Even a differential backup started at 6 p.m. could not be completed until 9 or 10 the next morning. With many users logging on at 6 or 7 a.m., open files (meaning they would not get backed up), slow backup and sluggish applications became the early morning norm.

"We really struggled to get any kind of a backup done," says Madewell. "We had to buy a lot of tapes, as well as two additional HP DLT tape drives to supplement our existing QualStar tape library."

SWS had been using Computer Associates (CA) software for years in its mainframe environment. So when CA organized a demo day onsite at SWS, Madewell made an appearance. One area of technology in particular made sense to him--storage resource management (SRM). He says one feature after another of CA's BrightStor SRM directly related to the daily storage nightmares he was experiencing.

"Centralized storage management, exception-based reporting throughout our network and capacity planning seemed like they were designed with our situation in mind," explains Madewell.

Before purchase, he loaded a demo copy on one server to test the product. He copied data to determine the amount of time that could be saved in server and storage management. Madewell then asked the SWS IT director to view the results before he approved the purchase.

LOW COST, QUICK RESULTS

Madewell says BrightStor SRM came in at less than $10,000--about $4,000 for software, $2,000 for maintenance and the rest for agents to place on nine servers used principally for storage. CA Services personnel came on site for two days to assist with installation and configuration.

Once installed, SWS immediately eliminated almost 25 gigabytes of data from nine servers. Backup times, too, have decreased markedly. A full backup now takes a day less and a differential backup is speeded up each day by a few hours. In North California, he reports, the amount of data backed up has dropped from 400 gigabytes to 300 gigabytes.

"As a result of the improvements in storage management, we haven't had a server down due to storage issues in almost a year," says Madewell. "Additionally, we haven't needed to add any more servers in order to provide user storage."

Reporting functionality, he says, is perhaps the greatest benefit to SWS. Reports are scheduled each night. When IT personnel come to work the next day, they can instantly view the state of storage resources and take action accordingly.

"Instead of reacting to the latest storage flap or downed server, we can now see what's happening ahead of time and be proactive about resolution," he offers.

By looking at reports, Madewell and his staff isolate offending corporate storage policies. The reports are forwarded to the department heads, who determine which files can be safely deleted. The next day, IT checks to verify the files are gone or takes further action.

CUSTOMIZATION PROVED DIFFICULT

SWS did encounter a few issues with BrightStor SRM, centered mainly around reporting and, in particular, the language used to write scripts-enterprise definition language (EDL). Madewell calls the basic reports that come with the product "quite good," but he required customization to find out what he really needed to know each day. The first elaborate script he tried to assemble took him almost 90 minutes. He had to rely on CA Services staff to author several scripts and teach him some of the basics of the language.

"The basic reports were a bit too high level for us, so we needed more granular data, such as file owner name and a list of the top 20 storage users," says Madewell. "As we didn't know EDL, the syntax was the biggest killer and learning it proved to be really challenging."

While help from CA Services on scripting and EDL syntax on version 6.2 of BrightStor SRM was needed, Madewell notes that version 6.3 comes with a scripting wizard to bypass the need to know EDL. He can now complete complex scripts in five to 10 minutes without error.

Another initial glitch concerned XP boxes. Madewell found running reports from XP clients difficult, a bug fixed in a recent service pack. That means he can now run reports from his laptop no matter where he connects to the statewide network.

Overall, though, Madewell says the solution has provided significant benefits. "I estimate $30,000 per year (in savings) by not having to purchase additional storage servers," he says. "That doesn't include the soft costs of losing two IT personnel for an hour or two each day, as well as a substantial amount of regained productivity throughout the company through greatly increased server uptime."

He also mentions that the company was about to buy a new backup system at a cost of $10,000. That purchase is no longer necessary.

"As much as 10% of our storage resources were eaten up in non-business and duplicate files," says Madewell. "Instead of manually searching each directory for rogue files, we can now monitor the proper usage of our entire storage environment from one console. We only have to add hardware when we truly need it."

SRM from CA

Computer Associates International Inc. (CA) helps manage the computing infrastructures of companies worldwide, including more than 95% of Fortune 500 companies. CA is a $3-billion revenue enterprise software company, providing software solutions that address all aspects of e-business management. Each is built on a common software infrastructure, can be used in a variety of mainframe and distributed environments, is integrated and platform-neutral.

Appointed in August 2002, Sanjay Kumar is chairman and chief executive officer of CA. He joined CA in 1987 upon its acquisition of UCCEL Corp., where he was a director of software development. Since then, he has held senior positions in development, strategic planning and operations, before his promotion to president and COO. Now a citizen of the United States, Kumar was born in Colombo, Sri Lanka, in 1962. He immigrated to the United States with his family when he was 14.

Under Kumar's guidance, CA redefined its focus with the introduction of a comprehensive technology strategy that aligned more than 1,200 products into a family of six brands: network and systems management (Unicenter); portal/business intelligence (CleverPath); application lifecycle management (AllFusion); security (eTrust); database management, application development and enterprise reporting (Advantage); and storage (BrightStor).

BrightStor Storage Resource Management and BrightStar Enterprise Backup manage all enterprise storage from a single point. This family of solutions maximizes existing IT and storage resources, integrates with different hardware and software, and can be scaled for businesses large or small.

BrightStor SRM is a software package that provides centralized management of storage resources for both distributed and mainframe systems. This cross-platform solution analyzes, manages, reports, schedules and automates networked storage resources across distributed and centralized environments in the enterprise. Comprehensive support exists for leading applications, databases, devices and systems to enable complete end-to-end resource management.

For more information from Computer Associates: www.rsleads.com/308cn-258