Communications and Data Management/Analysis Solutions for On-line Condition Monitoring of Substation Transformers Mark Cheatham Duke Energy Jim Dukarm Delta X Research Robin Thompson Sensei Solutions I. Introduction: Today’s substation Transformer asset manager is challenged to maintain the reliability and integrity of an aging and ailing fleet of assets subjected to increasingly higher demands while at the same time being expected to maintain or even reduce maintenance costs. Many major utilities have realized that with limited dollars and resources, employing on-line monitoring is a crucial component to being able to effectively and economically manage a large fleet of aging substation Transformers of which an ever increasing amount is reaching or will be reaching over the next decade, the expected end of useful life. With the deployment of on-line condition monitoring devices (e.g. on-line DGA, temperature monitoring, bushing monitoring, etc...) comes many challenges, most notably communications and effective data management/analysis. This paper will present the work of one utility working in conjunction with Delta X Research and Sensei Solutions to develop and deploy common platform communications and automated data management/analysis solutions for on-line monitoring of substation power Transformers. The paper will present the findings of pilot installations of Sensei communications and automated data management solutions for online DGA monitoring equipment deployed on substation transformers and the monitoring-data analysis capabilities and diagnostic alert functionality under development by Delta X research for TOA4. II. Condition Based Transformer Fleet Management and On-line Monitoring of Substation Transformers (On-Line DGA): Significant percentages of the US installed Transformer fleet have and will be reaching estimated end of insulation life over the next decade. It is believed that this will result in increasing future failure rates, to such a level that proactive measure in the form of monitoring with smart grid technologies (e.g. Online-DGA), refurbishment and replacement of Transformers or ancillary components, along with revisions to spare and mobile transformer levels will be required to effectively manage the aging transformer fleet.(1)

1

Financial constraints combined with the practice of demanding increasingly more from aging Transformers demands that more effective and proactive condition based management strategies are developed and deployed in order to maintain reliability and control capital replacement and maintenance costs moving forward. A key component of Condition Based Transformer fleet management strategies as noted above is employing the utilization of on-line monitoring, primarily on-line DGA. Dissolved GAS Analysis (DGA) is performed routinely on oil filled power Transformers to detect incipient faults and diagnose current state condition. DGA has become accepted by the industry as the most effective diagnostic tool for assessing equipment condition and providing the necessary information to assist the maintenance engineer in making appropriate disposition decisions for a given Transformer. (3) Many utilities perform DGA on critical substation transformers on an annual to semi-annual basis. The analysis has historically been performed by a technician visiting each Transformer and collecting a manual syringe sample from the drain valve of the in-service Transformer. The filled syringe is then sent to internal or external labs for gas chromatography which provides the breakdown of each of the gasses analyzed. The gasses measured in the analysis include key fault gasses seen in the table (3) below along with description of what the gasses indicate when detected in a Transformer. Table 1 Fault Gases (3)

Based on the levels, ratios, and rates of increases of the gasses noted above along with measurement of moisture in a given Transformer the maintenance engineer (Note. With years of experience in analyzing data) have the vitals of the asset and a key component necessary in assessing the assets condition and ultimately preventing potentially catastrophic failures via early detection of evolving faults. On-line DGA monitoring has been developed and continually evolving over the past several decades. Today the industry has several reliable and cost effective commercially available monitors capable of continuous on-line monitoring of the fault gasses listed above. The monitors available today range in capability for detection of one (typically; Hydrogen or composite), three (typically; variation of some hot metals, acetylene, hydrogen or carbon monoxide), and eight gasses (all gasses listed in table above), plus moisture. (2)

2

While manual collection of samples has historically served utilities extremely well over the past forty plus years, it is not without “faults” – pun intended... Manual sample collection relies on humans and thus is subject to human-error in the collection, handling, and lab-analysis of individual samples. If not performed properly by trained personnel it can result in potential for environmental impacts (i.e. spills of PCB contaminated oil) and even failure of the Transformer. Limited resources and constrained O&M dollars limits the frequency at which manual samples can be collected and analyzed thus reducing the chances of detection and ultimately prevention of potentially catastrophic failures of critical power Transformers. With the deployment of on-line DGA, Utilities have effectively brought the lab to the Transformer and provided the means to overcome the drawbacks and limitations of manual sample collection programs. It has been said “The Stone Age Didn’t End Because We Ran Out Of Stones” –Sheikh Yamani... with on-line DGA monitoring capability we have found a better way. With the advent and advancements in on-line DGA monitoring and the benefits it provides over manual sampling Utilities have a critical component necessary for the development of an on-line monitoring strategy to assist in the management of an aging fleet of critical substation power transformers. III. Communications and Data Management Challenges: Deployment of on-line monitoring comes with many challenges, most notably communications for remote data access and data management/analysis. Having reliable access to the data, proven analytics and diagnostic alert functionality are paramount to the effective utilization of on-line DGA technology. For many substations no pre-existing communications path for these devices being deployed on in-service Transformers has been established. Additionally developing effective data management and analytics for on-line DGA data is no trivial task. These challenges require decisions to be made around appropriate hardware options/requirements for communications, data protocols, data handling and storage options, and ultimately “now we have all this data, what do we do with it?”... There are many commercially available solutions to most of these issues; the key is finding the balance between reliability, effectiveness, cost, and meeting IT’s requirements. As a utility substation maintenance engineer overcoming these challenges can seem daunting when relying heavily on various internal organizational entities with limited resources to develop appropriate and scalable solutions for potentially varying on-line DGA monitoring technologies being deployed. Most of the on-line DGA devices currently deployed by Duke Energy have no direct communications to the device and rely on substation technician visits to extract data and in some cases alarms are wired out to local annunciator to give indication of a possible issue with the Transformer. Some monitors are equipped with cellular modems and are being monitored continuously via service provided by the on-line DGA monitor’s manufacturer. The ultimate goal with the deployment of this on-line DGA technology is to be able to rely on these devices to monitor DGA activity continuously and provide diagnostic alerts to 3

appropriate utility resources in a timely manner. The development of a common communications strategy with common scalable data management and analytics platform for the devices deployed and planned for deployment has been an evolving effort at Duke Energy over the past year. After several failed attempts to launch an internal solution Duke Energy set forth on a Pilot with Sensei Solutions and Delta X Research to develop and implement solutions to the communications and data management/analysis challenges described above. The goal of the Pilot was to select a single site currently equipped with on-line DGA monitoring - establish reliable communications for remote data access, automated data upload to Delta X Research’s TOA 04 web-based application for utilization of monitoring data analytics and diagnostic alerts functionality under development.

III. Pilot Installation: A. History of Pilot Location Table 2. Nameplate Information Transformer Nameplate Information Manufacturer – GE Rating – 50000kva Voltage – 105000grY – 13800 Type – FOA-T Impedance – 10.00% Serial number – 8567298 Designation – Bank 1B Gallons oil – 7800 Total weight - 190000 Date of manufacture – February 1950

This transformer is a generator step-up located at one of Duke Energy’s fossil powered facilities. Database records on the dissolved gas analysis date back to 9/25/1973. At that point in time the unit was exhibiting a thermal and/or partial discharge gas signature with increased 4

levels of CH4, C2H6 and H2. Subsequent sampling over time seems to suggest gas dissipation with the exception of H2. 9/25/1985 the combustibles re-surfaced with newly pronounced levels of CH4, C2H4 and H2. There was also at that time accumulations of C2H2 present, possibly indicating an active fault. Transformer was removed from service, repairs were performed with unit returning to service 11/1985 (there are no records on local file of the method of repair). Post-repair sample results were indicative of normal/healthy operation. 4/18/1994 the unit experienced a loss of all cooling. Thermal expansion of the oil caused an expulsion from the pressure relief device (stove pipe design). The transformer was taken out of service with testing performed. After completion of testing and repairs to cooling circuitry the unit was returned to service. Initial dissolved gas analysis following event did not indicate any abnormalities. 3/10/1995 gas generation had increased in levels of CH4, C2H6 and H2. There were also smaller increases in C2H4 and C2H2. Since that time gas trending experienced some fluctuation while maintaining concerning levels of hot metals and arcing gases. Decisions were made to oil process/de-gas the unit and install online DGA monitoring. This work was performed during the month of 10/2008. Immediately following was the install of the Sensei data acquisition and management solution described in the next section. (4) B. Sensei - Communications and Automated Data Management Solution The Sensei Solution is a modular and extensible platform for deployment of collaborative eventdriven knowledge-based process automation. Consisting of hardware, software and services, Sensei has delivered a complete solution to online management and monitoring of critical assets, including transformers, load-tap changers, breakers and relays. Architecture The Sensei Solution is comprised of two distinct components:

5



Sensei Master is the proprietary hardware deployed in the field to provide data integration and management of legacy devices, sensors and systems and provides the transport of those messages - the ultimate universal adapter - bridging the physical world of sensors and devices with the digital realm of networks, databases, and algorithms.



Sensei Net is the Enterprise Application software and services platform that supports multi-modal, multi-path data capture, normalization, transmission, storage and publishing to other enterprise applications and to corporate stakeholders through configurable user interfaces and messaging protocols. Collaborative analysis of data in a distributed, point to point or cloudbased topology o User-definable data visualization and reporting o User-definable condition-based notification o Integration of legacy information systems and databases o



CELOS is a scalable, standards-based cyber-security solution for management of access to hundreds of thousands of legacy devices and systems installed throughout the grid, in support of FERC and NERC mandates for cyber security compliance.

Software and Services Once the hardware has been installed and backhaul communication is established, the Sensei Master automatically downloads and installs all required configuration and application files, and commences processing. Sensei Net is complementary to and compatible with virtually all existing SCADA and WAN networks. The Sensei Master will share any available backhaul link, or if none is available, utilize Sensei Net wireless packet data service. All data collected by the Sensei Master is transmitted via secure encrypted link to it's designated Sensei Net host, where it is authenticated, validated, normalized, stored, and delivered to one or more destinations specified by the user. Data Destinations may be as simple as if-then scripts or may entail complex work-flow scenarios.

6

7

Deployment The Sensei Solution is deployed in two parts. First, a Sensei Master unit is installed at each remote site (both indoor and outdoor versions are available), providing the physical integration and communication point to which third-party devices and sensors may connected, utilizing any combination of Ethernet (802.3) or Serial (RS-232, RS-422/485, USB, PowerLine or wireless IP radio) interfaces. Second, backhaul connections from the Sensei Master to either Internet-hosted or Intranet-based servers are established using terrestrial (leased-line / dial-up) or wireless services. Sensei offers an inexpensive narrow-band wireless packet data service option, or customers may utilize any alternative wireless data service provider of their choice. Once received and authenticated, the data is normalized, stored, analyzed and delivered to one or more destinations, such as TOA4, MonitorWatch, Pi Historian, Cascade, Maxximo, or even Google Docs. The net result is secure and instantaneous communications, analysis and collaborative assessment of critical operational criteria and developing trends, from and between any device, system or user, in any format or protocol, hosted on any server, anywhere. All data driven and user defined. Because different devices and sensors present their data in a variety of formats and schemes, before the data can be shared with other users and applications it must be normalized. Normalization is the process whereby these disparate representations are unified into a standardized commonly accessible format (i.e. PPM vs. %, Fahrenheit vs. Centigrade, ASCII vs. EBCDIC vs. Binary Coded Decimal, BigEndian vs. LittleEndian, Integers vs. Real numbers, etc.) And because these data may be useful to more than one user or process, the data must be further translated to meet the requirements of the target data consumer. Translation (or Transformation) is the process of formatting select data elements in accordance with Destination-specific criteria, e.g. KELMAN or SERVERON MODBUS register set to TOA4 RPC post. C. Delta X Research - Development of Online DGA Data Analysis and Alerting Delta-X Research has been developing and marketing "Transformer Oil Analyst" software for management and interpretation of DGA data since 1993, and Duke Energy has been using that software since 1995. The current "production" version of the software, used by Duke, is an Internet web service called TOA4 Online. When multi-gas transformer DGA monitors were being developed in the late 1990's, Delta-X began looking at the problem of interpreting online monitor data and quickly discovered that new interpretive methods were needed, based on statistical techniques used for signal processing and time series analysis. High quality monitoring devices are now available, and the practical motivation for using them is very strong. Delta-X Research has been developing an extension to TOA4 Online called Monitor Watch to handle all kinds of online monitor data, including DGA, in a way that permits comparison of monitor-based results with the data obtained from laboratories and portable gas analyzers. 8

Although it may be barely possible for an engineer to inspect incoming laboratory DGA results and make decisions as to which transformers merit extra attention or need immediate action, the volume and the nature of data produced by online monitors makes automated data screening mandatory. Because the production and automated interpretation of the monitor data is a long-term ongoing process, there needs to be an automated means of attracting the engineer's attention in the rare instances when the results become "interesting," particularly if there are many transformers being monitored. It is desirable to provide early warning of real problems, but it is also highly undesirable to generate false alarms. It is not possible to eliminate false alarms completely without severely limiting fault detection sensitivity, and it is not possible to react to all faults instantly without suffering a high rate of false alarms. Finding an optimal balance of sensitivity to incipient problems on the one hand, versus tendency to produce false alarms on the other hand, requires the application of appropriate statistical methods. To support condition-based asset management, automated processing of online monitor data needs to meet these basic objectives: 

Early warning of unexplained suspicious changes



Ongoing assessment of equipment condition



Ongoing assessment of monitor condition



Provide insight into typical data patterns

It should be noted that an "early" warning may not be "instant." Online monitors are not real-time devices -- samples are at least one hour apart and typically are four to six hours apart. Monitor data can be noisy or intermittent. Except for the most extreme changes, it may take several samples (sometimes amounting to a day or two) before a departure from typical patterns can be identified with high enough confidence to warrant an automatic alert. Without this latency, false alarms could be frequent. The amount of latency required depends on several factors:   

How much tolerance there is for false alarms Subtlety of the change is that is being detected Efficiency of statistical methods used for detecting changes

"Unexplained suspicious changes" above refers to the fact that alerts should only be raised in response to changes, not steady-state conditions and not transient events such as a spike in a single gas. Only those changes which could be fault-related and are not clearly related to something other than a fault should raise alerts. For example, when the ambient temperature rises, the combustible gas concentrations in the oil will invariably rise a little, only to decrease 9

again when the temperature declines, so in this case a coordinated rise in combustible gas concentrations should not raise an alert. For condition-based asset management, it is valuable to have continuous awareness of the condition and serviceability of the transformer and (if possible) its associated bushings, tapchangers, and arrestors. Monitoring of a healthy transformer provides a good understanding of its typical responses to day-to-day operating conditions and allows the assessment of its capacity to withstand and recover from exceptional load and stress, provided that the monitor data are continuously collected and made readily available for investigation. When a problem is detected, the automatic data interpretation must include an estimate of severity. Graphical visualization of data is essential for human understanding and verification of the condition assessment provided. Since the monitor is itself a complex machine exposed to the weather and to heat and vibration from the transformer, it is useful to pay attention to whatever self-check data the monitor provides and also to watch the measurement data for excessive noisiness, spikes, dropouts, and other conditions that may indicate that the monitor needs attention. For the GSU monitoring pilot project, a Sensei Moto data acquisition device was programmed by its manufacturer, Sensei Solutions, to poll the GSU's on-line DGA 8-gas monitor and transmit the data over a secure wireless link to a Sensei Net server which, using a secure protocol, retransmitted the data to TOA4 Online running on its server in a large data center in Texas. Since Monitor Watch at that time was not yet installed on the server, the monitor data was downloaded daily from TOA4 Online and processed by Monitor Watch running on an office computer at Delta-X Research. The resulting monitor status report, in the form of a web page, was then posted on a web server accessible to project personnel.

10

The monitor status report provides a summary of the transformer's condition as of the latest sample. Figure 1. Monitor Status Report

In this case, the available indicators of transformer condition are:     

DGA result (condition code: 1 = OK, ..., 4 = severe problem) O2/N2 ratio (indicates whether dissolved oxygen and nitrogen are in expected proportions) CO2/CO (indicator of cellulose degradation by faults) Moisture code (1 = OK, 2 = high, 3 = very high) RS% baseline (95% confidence interval for long-term average relative saturation)

The software has highlighted the moisture code (2) to call attention to the fact that the percent relative saturation baseline, which is an indicator of the active moisture content of the cellulose insulation, is above the caution limit of 20%. Below the equipment condition summary is a text area indicating which monitored variables are currently exhibiting potentially interesting, although not necessarily abnormal, behavior. Both CO and CO2 have recently experienced trend changes; several variables show generally upward trends; methane (CH4) has a high level of "noise"; and acetylene (C2H2) has had a high percentage of recent "null" or missing values. Below the summary information is a graphical display area which provides details of the current status of individual monitored variables. For example, here is the methane status:

11

Figure 2. Graphical Display Example 1 (CH4)

The chart shows a steady mild upward trend (blue line) of approximately 0.04 ppm/day over the last 90 days. The jagged gray outline is a plot of the methane values reported by the monitor. It is easy to see that individual values cannot be taken seriously. In particular, the simple-minded application of a fixed ppm limit to this kind of data would tend to produce frequent false alarms. The percent relative saturation status shows that although the %RS baseline is marginally above the caution limit of 20%, the general trend of %RS is close to 20% and moving slowly downwards, so there is no indication of worsening cellulose moisture. Figure 3. Graphical Display Example 2 (%RS)

Much of the "noise" in the %RS data consists of bumps correlated with temperature changes, so the 15.3% relative noise indicated here is not a sign of poor measurement precision. In the equipment status summary there is a link to an event log which records the date and time of notable changes. For example, the date and time when the moisture code switched from 12

1 to 2 because of %RS is given, and then later the date and time when a gap of 1 day and 11 hours in the data was noted following an interruption in monitoring. Figure 4. Equipment Status Summary Output Example ============================================================= 2009-02-07 20:02:00,MOISTURE,1|2,High RS% baseline,c2h2? relsaturation~ airtemp?

2009-02-17 20:02:00,GAP,,,1 d 11.000 h

========================================================= During the time from early February to mid-May 2009 when the monitoring pilot project was underway, the monitor status reports showed consistently that combustible gas levels were low and in some cases slowly increasing, as would be expected in a transformer that was recently degassed. The CO2/CO ratio, although marginally above its customary upper limit of 10, had a horizontal trend, as did the O2/N2 ratio. Although there was some indication that the cellulose moisture was marginally above its caution limit, the %RS trend showed that this too was not new and was not worsening. Aside from the analysis and reporting function, the software must also provide automatic alerting. That capability is already present in TOA4 Online as an optional feature called TOA4 Diagnostic Alerts. Based on a scheme used by another large electric utility customer of Delta-X Research, Diagnostic Alerts is very different from real-time event notification systems. Every few hours, Diagnostic Alerts scans recently received data and compiles a summary, for each user, of transformers and other equipment which -- according to criteria specified by the individual user -- has raised an alert. Any "alerted" equipment is added to the user's Alerts List in TOA4 Online, where it stays for 90 days or until removed by the user. While on the Alerts List, an equipment item cannot raise further alerts for the same kind of problem (e.g. DGA), since the user is already aware that the equipment needs attention. The user can choose to have new-alerts summaries delivered by e-mail similar to this example.

13

Figure 5. Diagnostic Alert Example ============================================================= From: TOA4 Diagnostic Alerts

Subject: TOA4 Alerts for pshort-r004 in @PSHORT (1 new alerts)

To: [email protected]

Date: Mon, 13 Jul 2009 19:10:02 +0000 (UTC)

Designation

Equipment

Alert

Sampled

-----------------------------------------------------------------

HNT T4

TRN E0589

MONITOR DGA

2009-06-14



-----------------------------------------------------------------

============================================================= In the next phase of testing, Monitor Watch is being installed on the TOA4 Online secondary server, where it will be accessible for direct data uploads and status retrievals by data acquisition systems such as Sensei's. In some scenarios to be tested shortly, a "data historian" database will be a relay point between the monitor and Monitor Watch. On the server, Monitor Watch will function automatically. Duke Energy users will be able to log in and view a variety of monitoring reports, including equipment status, equipment comparisons, exception reports, and comparisons with laboratory data. Users will also be able to download monitor data for spreadsheet-based investigation. For this phase of testing, Monitor Watch will be integrated with TOA4 Diagnostic Alerts.

14

IV. Conclusions: After several failed attempts to launch an internal solution Duke Energy set forth on a Pilot with Sensei Solutions and Delta X Research to develop and implement solutions to the communications and data management/analysis challenges described above. The goal of the Pilot was to select a single site currently equipped with on-line DGA monitoring - establish reliable communications for remote data access, automated data upload to Delta X Research’s TOA 04 web-based application for utilization of monitoring data analytics and diagnostic alerts functionality under development. The Pilot accomplished all of the Goals outlined above and was implemented in a timely (less than a month after project kick-off) fashion that did not require the development of any custom software by Duke Energy. Having reliable access to the data, proven analytics and diagnostic alert functionality are paramount to the effective utilization of on-line DGA technology. This Pilot provided Duke Energy with a potential solution toward meeting these core principles for On-line DGA and has shown the potential to be expanded to provide solutions for other on-line monitoring (including Transformer temperature monitoring data, bushing monitoring data, etc…). Works Cited (1) Bartley, W. H. (1997). Keeping The Lights On: An An Action Plan For America's Aging Utility Transformers. (2) CIGRE, L. M. (2002). Guidelines For Life Management Techniques For Power Transformers. (3) Jakob, F. (n.d.). Disolved Gas Analysis - Past Present and Future. (4) Jackson, Thomas C. Substation Transformer Tech Support Interview

15