Case Study

Santos Beats “Bandwidth Bracket Creep”

An application performance monitoring and management system from South Australian software developer Foursticks has become a major operational cost saver for leading Australian energy company Santos Limited.

An oil and gas producer with assets of over A$ 5 billion and annual oil production of around 55 million barrels, Santos has interests in every major Australian petroleum province.

The company supplies gas to all mainland Australian States and Territories and sells oil and liquids to a number of domestic and international customers.

The company depends on its wide area network (WAN) and Telstra’s IP network for communication between its headquarters in the South Australian capital, Adelaide, other State capitals, such as Brisbane and Perth, and site offices in rural areas of Australia. Communications to major field sites such as Moomba and Ballera and numerous smaller field sites are carried over private microwave bearers. It also operates a virtual private network (VPN) to Jakarta in Indonesia, and Houston, Texas.

IT systems are managed on behalf of the company by the IT Services Department, based at head office in Adelaide.

The Problem

Before Foursticks, the Communication Services team was frustrated by its lack of ability to demonstrate and control how bandwidth was being used throughout the company.

“In the environment we were operating in we were facing inevitable bandwidth bracket creep,” said Communication Services Co-Ordinator Lloyd Binns.

“The reaction to any application response issue was to demand more bandwidth and without any evidence to the contrary it was a hard demand to counter.”

A 4MB service between Adelaide and Brisbane, was heavily used for a wide variety of IT purposes. It was an expensive pipeline and a particularly troublesome link.

“Users were frequently complaining about application response times,” Mr Binns said. “We believed this was because a lot of low priority and ‘bursty’ activity was slowing down critical business applications. But, because we couldn’t graphically demonstrate it, we came under pressure to double the link to 8MB at considerable added expense.

Net Prioritizer (NP) at Santos

The issue starkly highlighted the need for a way to “get inside” the network to view the source of problems and gather the facts on which to base procurement decisions.

“The other side of that coin is awareness of what you’re doing and better management of what you have,” Mr Binns said.

The Solution

In late 2002, the Santos Communication Services unit started looking around for a solution.

Mr Binns heard around the traps that Foursticks was doing “marvellous things with bandwidth monitoring”

The name Foursticks stuck in his mind and when Foursticks called in, and we were happy to discuss what they had to offer”, he said.

That turned out to be Foursticks NP Gateway, a software solution that could be used on regular commodity servers to provide an all-in-one monitoring and policy-based management service.

NP Gateway does this by automatically giving critical applications as much bandwidth as necessary for smooth operation in real time, while optimizing the performance for remaining services.

The Implementation

With its product newly released on to the market, it was early days for Foursticks. The company worked closely with Santos to overcome some teething problems.

Because Santos supports its own infrastructure, it has a much higher level of in-house networking expertise and selected to set its own policies.

“The product was still evolving and we were all learning,” Mr Binns explained. “The Communications Services team decided to do most of the work to develop in-house skills, and that initially slowed us down. But once we got serious about creating a representative set of policies as a default, it turned out to be less than a day’s work for 20 sites.

“Foursticks’ technical support was invaluable – they would be on call as soon as we had a headache of any description. But once you understand the logic of what you’re doing and someone can explain the topology of the network, using NP Gateway is not a great mind bender.”

The Results

NP Gateway began to show its value in numerous ways, some of them unexpected.

For a start, Communication Services noted a dramatic decrease in performance related complaints over the next few months.

When there were complaints or demands for more bandwidth, the Coms engineers were able to demonstrate easily and quickly that application response was affected by the way the network was being used, rather than the overall bandwidth available.

“Even better, we were able to reduce the Brisbane link from 8MB to 4MB, saving a very significant amount in bandwidth rental costs,” Mr Binns said.

Significantly, Foursticks NP gave the team unprecedented visibility of the network, the ability to diagnose problems quickly, and an automated, real-time way to solve them.

There were a number of occasions when bandwidth increases were avoided simply by having the awareness provided by the real time, 1 second granularity bandwidth usage reports

NP in Action

“It’s hard to measure how often Foursticks NP helps us with traffic management and I sometimes think it would be great if a light flashed on the box when policies were being exercised,” Mr Binns says.

Here are a few examples of NP in action.

Santos staff at a remote satellite pumping station were experiencing slow application response times, and placed a Service Centre call.

After using Foursticks NP to diagnose the problem, Communication Services found that a particular laptop was generating a continuous 400k bps stream of traffic.

“We had detected a rogue application that was trying to connect to Adelaide automatically and flooding the WAN,” Mr Binns said.

“The user didn’t even know it was running. We then implemented a policy on NP to limit the impact on business related traffic while waiting for the user to return from the field.”

On another occasion the IT team used Foursticks NP to confirm that an increase in Internet bandwidth up to 10Mb was sufficient for an investor web cast. The web cast peaked at 7Mb. NP may be used in future web casts to better synchronize audio and video streams.

NP policies don’t just guarantee application response times, they allow Santos to deliberately hold back intrusive applications such as Microsoft operating systems updates and patches.

“NP led us to discover how much the updates were impacting on service and how we could limit the impact,” Mr Binns said.

“NP is a very flexible tool. It has even helped us confirm whether we had eliminated all of an Internet worm virus.”

In general, the Communication Services team has been freed to focus on strategic business issues and pursue the company vision of relentless improvement and customer service.

“Internet bandwidth is another area that was experiencing uncontrolled growth and we didn’t know why,” Mr Binns said.

“Now we can use just as much as we need and service our overseas VPNs more effectively by prioritizing our applications in front of email, for instance.

“We can zero in on each site and look at bandwidth use and any limitations, then provide decent communications without extra cost.”

Future Plans

Santos Communications Services team has a goal to further reduce Santos’s bandwidth spend by 10 percent this year using the insight that Foursticks gives to align bandwidth supply with actual demand.

“Some of our initial ideas about what we were going to do with Foursticks haven’t transpired,” Mr Binns said. “The whole network model has changed due to business forces.”

Foursticks NP will be used to facilitate some of these strategic changes, which include a plan to link Oracle with major external customers and suppliers using web services.

“With major projects like Mutineer/Exeter offshore from Western Australia we’ll be turning our thoughts to Foursticks to manage the Perth bandwidth and bringing in new international links which aren’t cheap, so we’ll be watching that too,” Mr Binns said.

“We are now taking our application management to a new level by using a bandwidth and latency generator appliance that will sit between two routers to measure exactly how much bandwidth an application really does need,” Mr Binns said.

“This information will then be extrapolated to create policies on the Foursticks system.”

For IT industry veteran Lloyd Binns, Foursticks NP has been a good news story. “I’ve been in computers for 35-odd years, and for all of that time there’s been an underlying assumption by users that all application performance issues were related to lack of bandwidth,” Mr Binns said.

“I’ve brought people into my office to watch the NP Console for a while, and they usually go away understanding in layman’s terms the difference between bandwidth or throughput and speed or latency. There is now a much broader understanding of the factors affecting application performance.”

Santos’ Communication Services team has set default policies to ensure unsanctioned traffic has zero impact, and sets new policies to guarantee service each time a new application is implemented.

“We have a better understanding of how the network’s used and that’s reduced costs and stress all round. Foursticks NP has empowered us to confidently respond to changing business needs.”