Process Automation

With limited expertise available, most companies are only in early inventory gathering stages, with lots of work ahead of midnight, December 31, 1999 deadline

Y2K Computer Compliance Crisis Builds on Pulp and Paper Mill Operating Floors

BY KEN L. PATRICK,Senior Editorial Consultant

A crisis is quietly building IN the pulp and paper industry, as well as in most other businesses and industries worldwide. As the year 2000 ratchets nearer one day at a time, the timetable and options for Y2K computer compliance narrow proportionally. Simply stated, at midnight on December 31, 1999, many pulp and paper mills will not have made computer adjustments necessary to prevent major process shutdowns and information control chaos.

Generally, in the race to Y2K compliance, the pulp and paper industry is not in good shape. Consultant companies tracking the problem are beginning to issue warnings that it's getting to be later than most people realize, and that there won't be nearly enough computer expertise available to handle the industry's needs, particularly in desperate, eleventh-hour situations.

At its core, the Y2K crisis is almost humorously simple-a limited two-digit date field that does not allow the next millenium to happen, a problem caused by lack of foresight beginning several decades ago but still being propagated today. During the past 30 years, most computer systems were designed to represent century dates with two digits only (e.g. "1999" is represented as "99" with the "19" being assumed). If computers interpret the year 2000 as solely 00, errors in date representation and related calculations can occur. Some of the reasons behind two-digit date field problems are listed in Table 1.

The problem is complicated because dates are used in subtle and unpredictable places, including those that may be imbedded in obscure computer system logic. Table 2 reflects the variety of date representations that have built up during the past quarter of a century. Date-related misinterpretations can cause computer system failure for several reasons.

For example, the system can misinterpret the "00" as "1900," a malfunction can result because of missing data, data can be sorted in an erroneous fashion, or a shutdown can proceed because the date was used in an arithmetic calculation (a division by "0" occurs, as discussed in an example below).

But although the root cause may be absurdly simple, Y2K complications can be far-reaching and potentially devastating, especially on the mill operating floor. They can affect data files, screens, utility programs, packaged applications, reports, interfaces, operating systems, control languages, parameter libraries, naming standards, hardware, etc.

Also, areas not normally thought to be affected can be literally shut down, such as telephones, elevators, security, and lighting systems. The solutions to these problems can become very involved and time-consuming.

INDUSTRY AWAKENING. Tom Bruhn, manager of business development--automated systems at Raytheon Engineers & Constructors, a company that has been gearing up for Y2K compliance consulting for some time now, points out that there is an "awakening" in the paper and other industries, but that most companies are just now in the first inventory gathering stage. In fact, he estimates that as much as 95% of the big companies, in all industry areas, are just now in this beginning stage.

"If you drill down into what's going on at most companies, you'll typically find that individual plants are responsible for handling their own compliance problems," Bruhn explains. These plants, he adds, are usually asking people in the various process areas to come up with inventory items, and are compiling these in Excel spreadsheets, etc. The format they come up with, however, might be radically different from that at another mill or plant in the same company. The upshot is an inventory hodgepodge in many different formats.

Key ingredients in any Y2K compliance plan, Bruhn emphasizes, is the development of a consistent, accurate inventory of all control devices at risk, and the maintenance of this inventory in a reliable repository. Raytheon has put together a detailed, highly organized approach to Y2K that not only includes these ingredients, but takes compliance through several other critical stages to a manageable conclusion in the fastest possible time and at minimum costs. Raytheon is currently working with several major pulp and paper companies to implement the approach known as "How to 2000."

According to Mark Cubine, director of business development--automated systems at Raytheon, most of the larger pulp and paper companies have had a Y2K program in place for a year or more, and have already begun to address the information technology side of the problem-general financial ledgers (accounts receivables, payrolls, etc). Across all industry lines, however, these people tend to be MIS-IT (information technology) types who sometimes don't have in-depth knowledge and understanding of what the needs are on the mill production floor.

In Bruhn's view, there often is a "firewall" of sorts between the IT and production segments of an industry that slows, if not prevents, communications. So, in cases where a plant is responsible for its own Y2K program, two things tend to happen. First is a type of "denial," where a plant grossly underestimates its exposure. The plant will conduct its own inventory and find perhaps three or four systems at risk. In reality, of course, there are usually many, many more tie-ins, Bruhn explains, that are simply overlooked. The tendency is to not include these smaller items in an inventory, when, in fact, "any one of them could shut down a process."

He further explains that in the pulp and paper industry, as in most all industries, there is an awareness in the IT ranks that something also needs to be done on the shop floor, that there could be problems down there as well. "And they all say that something is being done," he points out, "but the approach is typically haphazard because the operators who most often are given the responsibility of conducting the program don't really have any formal Y2K training. Their judgments are understandably going to be subjective and biased, based on the experiences they have."

To this, Graham Jaques, business development manager for Raytheon's automated systems division in London, adds that the pressures of just keeping a mill running are enormous, and that everything else becomes more or less secondary to that: "Everybody has other things to do." He also emphasizes that a letter of compliance companies might get from a controls manufacturer can give them a false sense of security. "This can be frightening," he warns. "Just having some files on the shelf doesn't solve the problems."

According to Cubine, "It's not that most of these problems are so technically serious or unsolvable. It's just that you have to know the exact nature of them to carry out a workable solution. Actually, the whole compliance thing is a matter of risk management," he states.

As an example, Cubine and Bruhn point to a case involving a turbine generator. The system used a pulse train temperature sensor reporting back to a programmable controller. The controller was counting pulses between two time stamps and assigning a temperature value. In doing a Y2K rollover test on the system, Raytheon found that the assigned temperature value went to infinity at the year 2000, which immediately started the shutdown procedure.

"This is a perfect example of the importance of knowing exactly what day and time it is. The control system was using today's date minus whatever date to determine the temperature value. When suddenly one of these goes to zero (in a two-digit date field the year 2000 will typically show as 00), the program is in a position of having to divide by zero, which results in infinity. It doesn't make much sense to the controller," Bruhn explains.

Table 3 examines 12 common problems the Raytheon group found occurring repeatedly during Y2K rollover tests they have been conducting in a variety of industries, including pulp and paper. Possible solutions are listed for each specific problem.

THE TROUBLE WITH TRIBBLES. Jaques points out that a major problem, especially at paper companies with integrated product conversion lines, is the extensive use of controllers supplied by relatively unknown, small family-type firms or systems integration houses.

This is particularly acute in Europe where mills tend to have major conversion and packaging lines following the paper machine. "They might be able to produce jumbo reels, but Y2K failure of any of these smaller controllers can bring production to a standstill. Even elevators and conveying lines can go down at midnight when the millenium changes-if the proper adjustments are not made," Jaques says.

He emphasizes that when dealing with the larger controls firms, companies will be fairly secure. But many of the small, third-party, "brother-in-law" types of operations are literally untraceable. Some of these small operations have gone out of business, many have merged with other companies, others have changed their names and product lines, etc. Acquiring information on a system supplied even just a few years ago can be difficult if not impossible.

One pulp and paper mill, Cubine tells, recently used an outside group to conduct an inventory. A major problem developed around the fact that those conducting the inventory really didn't understand the process and didn't know the equipment and control system suppliers very well. "We came in behind this study, and found that many of the vendors, for example, were determined simply by taking names from the OEM control cabinets. And, of course, in most cases the OEMs don't manufacture these controllers. We opened one of these cabinets and found 64 different components, which had to be individually traced back through the respective suppliers."

There can be many problems in these OEM setups, and the only way to really determine what and how severe the problems might be is to test. "However," Cubine stresses, "the number of available testing days-which implies downtime-between now and the year 2000 is very limited. When you look at a mill's calendar, how many planned downtime days are there between now and the year 2000-20 to 30 days perhaps? Certainly, a mill won't be able to test while it's running."

Bruhn says that, also, a mill might not have electrical power available for testing during a planned shutdown. "Some kind of testing power will be needed during these shutdowns. That's why a good methodology and a well-planned approach is so important. You have to do a lot of advanced planning and have a high level of awareness. You have to look very closely at all of the issues involved and understand how the process unfolds to figure the logistics and make the proper determinations."

Typically, a mill will have a proliferation of small packaged control systems in the environmental area, wastewater treatment plant, mill water supply, etc. Some of these, Cubine explains, will have been supplied by "fly-by-night" vendors and some will have been installed by "mainstream" suppliers. "In recent years, many companies have installed programmable controllers with HMIs (human-machine interface devices) on the front end. Certain versions of these are not Y2K compliant and will have to be fixed, which in most cases will be relatively easy.

"But when these devices fail, the result can be catastrophic. Mills no longer have knobs, gauges, and dials for manual control. These were done away with some years ago. Now, mechanisms along virtually every step of a process work back through the controllers. So the only option is to fix these, i.e., bring them into Y2K compliance," Cubine emphasizes.

HOW TO 2000. Raytheon's approach to Y2K compliance breaks the process into two basic stages-detection and compliance. It begins with a "planning and awareness" step in which compliance is clearly defined for a specific mill or operation. Awareness is built up in the various levels of management, and budgeting for the project is approved. An "enterprise schematic" is developed, graphically representing the mill's control system environment in great detail.

Following initial planning, staffing, and funding, a logical inventorying of the mill's automated systems begins (Figure 1). Although the inventory is a critical step in the process, Bruhn again emphasizes that it is only the very beginning. This is followed by a data store or "year 2000 repository."

The repository is the key to getting all information into a logical format so that data can be thoroughly analyzed. Raytheon's Y2K database in Birmingham, Ala., is a master repository of "everything we've ever looked at and every encounter we've had with every customer we've ever worked with," Cubine points out. Currently the database has 12,000 to 14,000 items, which probably will grow to 100,000 items by the end of this year, he adds.

In the next "triage" stage, the inventory is pared down and prioritized. A priority is assigned to each system component based on business impact, cost, and expected time to repair. Some components will not be affected at all, and these can be eliminated from further Y2K assessment, reducing project cost and scope. Others may be found to be noncompliant but not serious enough for highest priority action (below the risk factor threshold). Bruhn describes the main purpose of this stage as defining specifically those systems that a mill will have to worry about-those that can and will definitely "hurt" a mill.

Three options then follow the triage stage. In the first category go those units determined to be Y2K compliant. The only action needed with these units is to update the database repository. In the opposite category are devices found to be noncompliant with known problems (from information in the database or gathered elsewhere). In the "between" category are items whose compliance is either unknown, or they are known to be noncompliant but the exact problems are unknown.

A detailed assessment must be conducted on those devices in the "between" compliance-unknown category. It will be necessary to determine the status and exact problems associated with devices in this category through direct testing, follow-up with vendors, etc. After assessment, these devices are either reinserted into the triage stage (found to be compliant) or reclassified as noncompliant with known problems.

Some important choices will have to then be made regarding items in the noncompliant (with known problems) category, which is primarily a consultant issue, according to Cubine. Basically, this involves a repair, replace, or retire decision based on "what's out there that could be implemented, cost, the time frame available, etc."

Some key strategic decisions have to be made at this stage. If the unit is to be repaired, there will need to be a detailed plan: how will it be done-onsite or offsite (at a remediation facility, etc.)? If it's to be replaced, what will be the schedule and what resources will be needed? If it is retired, what will be implemented to take its place-who will write the manual procedures, etc.? "This is a detailed process that requires someone who has been at the end and can look back and give directions," Bruhn stresses.

Handling solutions in the noncompliant with known problems stage is the "heart" of a Y2K program, according to Cubine, Bruhn, and Jaques. Raytheon anticipates that some 20% to 25% of the pulp and paper industry's day-use systems will be found to be noncompliant. The biggest item, the DCS, will probably be about 85% compliant, and where there is noncompliance, the vendor can fairly quickly and easily fix the problem. In the programmable logic controller (PLC) area, about 95% are expected to be either compliant or compliance doesn't matter. And the 5% fix will generally be very simple (replacing a real-time clock, etc.).

But the general PC applications area, including the myriad of packaged control systems installed on OEM equipment, in the environmental sector, being used for performance monitoring, etc., is expected to have a very high level of noncompliance-as high as 50% or more.

"Assume that 30% of a mill's total system is in the noncompliant-with-known-problem category. Further assume that the mill will be repairing maybe 85% of that 30%, and of that 85% say that 25% to 30% do not have traceable vendors. These will have to be fixed. It will be necessary to go in with a source code, and in some instances there will be no source code.

"A decision regarding the level of compliance a mill plans to attain should have been determined much earlier in the program. If the decision in case of a bad code is not to use any quick-fix/ "trick" techniques such as "windowing" or moving dates, then the only approach is to change a two-digit field to a four-digit field, which is complete or ultimate compliance. If so, this will involve a completely different skill set from that used in the earlier stages of the program.

"Also, changing a natural code adds new problem risk. Some testing will be needed to assure that, although the device appears to operate O.K., its functionality has not been altered in any way that will affect other systems it is dependent on or with. It gets very detailed and involved," Bruhn explains.

The actual definition of "compliance" will vary. Sometimes it might mean full four-digit compliance, but often alternatives such as windowing (masking), offset representation (using diversion programs), encoding (compress/decompress), etc., might be better choices in some mills, again depending on the degree of compliance it plans to achieve.

The final stage in How to 2000 is called "fallout." In this stage, a mechanism is put in place to be sure that compliance problems don't recur or that new problems aren't created. With the fallout stage in effect, every new control that comes into a facility must pass through "the eye of a needle" to be sure that it is fully compliant before it is accepted and installed.

The fallout stage includes procedures and policies for procurement, testing, and acceptance testing. "We're not going to allow any new problems to creep in," Cubine asserts. He points out that, surprisingly, companies are still shipping control systems and devices today that have the Y2K compliance problem. And, he adds, they will continue to do so for several more years.

TIME GROWING SHORT. The timetable for a mill or plant to achieve compliance obviously depends on that facility's process and specific control setups. But, generally, Cubine explains, the inventory stage will require about a month or less. The triage and detailed assessment stages might require another two months or so. The correction stages, depending on many, many variables, could be relatively short at six months, but could extend much longer.

"There is no 'magic bullet,' Cubine, Bruhn, and Jaques repeatedly warn. "The problem is much bigger than most people anticipated, and the resources available to handle it are finite. Some businesses and industries are simply going to come up short.

"Those who think they can wait until the last minute are in for a big surprise. Although some companies such as Raytheon are set up now to handle the problem, they will be reaching a collective maximum load at some point in the near future. Now is the time to begin forming Y2K compliance partnerships. Six months from now might be too late," they emphasize. n

KEN L. PATRICK, senior editorial consultant for Pulp & Paper, is president of paper Industry Communications Inc., Atlanta, Ga.