Optimal Capacitor Placement in Radial Distribution Systems by Reinforcement Learning Approach D. ISSICABA† , A. L. BETTIOL‡ , J. COELHO† and M. V. P. ALCANTARA§ †

Electrical Systems Planning Research Laboratory, Department of Electrical Engineering Federal University of Santa Catarina Trindade, Florianopolis, SC, PO. BOX 476, CEP 88040-900, Brazil [email protected], [email protected] ‡ Department of Electrical Engineering SATC Faculty Pascoal Meller 73, Criciuma, SC, CEP 88805-380, Brazil [email protected] § Goias Energy Company Jardim Goias, Goiania, GO, CEP 75805-180, Brazil [email protected]

Abstract: - Inherent to power transmission and distribution are the electrical power losses. These electrical losses can be considerably reduced through the installation and control of reactive compensation equipments, such as capacitor banks, reducing reactive currents in distribution feeders. This paper presents a new approach for optimization and automatic control of reactive power in distribution systems, based on a reinforcement learning algorithms and sensitivity analysis. An optimal capacitor placement, sizing, and controlling problem is formulated aiming to improve voltage regulation and reduce power losses. Characterize and compose the main contribution of the work, a formulation and methodology to a concomitant search for optimal placement, optimal placement policy, and optimal control scheme of capacitor banks in distribution systems. These optimal solutions, provide decision support to reactive power compensation planning in large scale energy utilities. The proposed method has been tested in a Brazilian Central Region real system with preliminary but promising numerical results. Key-Words: - Capacitor placement, Loss minimization, Distribution planning, VAR support, Reinforcement learning, Machine learning.

1 Introduction Electrical power losses in distribution systems correspond to about 70% of total losses in electric power systems [1]. These electrical losses can be considerably reduced through the installation and control of reactive support equipments, such as capacitor banks, reducing reactive currents in distribution feeders. Furthermore, voltage profiles, power-factor and feeder capability of distribution substations are also significantly improved. Computational techniques for capacitor placement in distribution systems, have been extensively researched since the 60’s with several available technical publications in this research area [2]. These publications describe several approaches and techniques to the problem, standing out the analytic methods [3], [4], heuristic methods [5], [6], numerical programming [7], [8], fuzzy logic [9], [10], ant colony optimization [11], [12], tabu search [13], [14], neural networks [15], genetic algorithms [16], [17] and hybrid methods [18],

[19]. Compelled to identify the location, number, size, type, and control scheme for each capacitor to be installed in a distribution system, the problem is usually formulated in terms of a combinatorial optimization problem, where conflicting objectives are considered as purchase and installation cost minimization of capacitors banks and electrical losses reduction. Despite quality and quantity of works on the issue, established a final outcome and, due lack of human and financial resources, electric utilities usually implement gradually intermediate non-optimal solutions to the problem. In addition, it’s a common practice, especially in electric companies with large concession areas and very long feeders, to apply these algorithms in planning scenarios studies, regarding different financial constraints represented by number (or size) limits to capacitors banks at buses, feeders and/or distribution substations. Nevertheless, each budget constraints de-

fines a new combinatorial optimization problem and, not rarely, these solutions might demand reactive power compensation equipments, unfeasible on a strict technical optimal placement solution without budgets constraints. This paper proposes a new approach for optimization and automatic control of reactive power in feeders and substations of electric power distribution systems. Characterize and compose the main contribution of the work, a formulation and methodology to a concomitant search for optimal placement, optimal placement policy, and optimal control scheme of capacitor banks in distribution systems. The methodology uses reinforcement learning concepts and algorithms, as well as bus sensitivity-based analysis with respect to reactive power injections. These optimal solutions, provide decision support to reactive power compensation planning in large scale energy utilities. The proposed method has already been tested in a Brazilian Central Region real system with preliminary but promising results. The paper is divided into five sections as followings. Sections 2 and 3 present, a brief introduction to reinforcement learning paradigm and a description of proposed approach, respectively. Section 4 shows some preliminary numerical results. Finally in section 5, main conclusions and future research perspectives are outlined by the authors.

Nomenclature f (v, z) fC (v, z) fP (v, z) v z vkm zkm Ckm vm zm Cm PLm κm L κf,s C

Cost function Cost of fixed and switched capacitors Cost of electrical losses State vector of voltage magnitudes Capacitor switching boolean vector Voltage magnitude at node k for load level m Boolean variable denoting capacitance existence at node k for load level m Size of the capacitors connected at node k for load level m Voltage magnitudes for load level m Vector of boolean variables [zkm ]N Vector of capacitor sizes [Ckm ]N Power losses for load level m Cost of energy losses ($/MWh) for load level m Fixed or switched capacitance cost per unit of size C ($/kVAr)

PLm κm L κf,s C Ckt,Θ Tm g(v m , z m ) V min V max β N , NL Λ, Θ i Zab

Power losses for load level m Cost of energy losses ($/MWh) for load level m Fixed or switched capacitance cost per unit of size C ($/kVAr) Capacitor sizes at node k for Θ load levels Time duration for load level m (h/year) Power flow constraints Lower bound of nodal voltage magnitudes Upper bound of nodal voltage magnitudes Penalty factor Number of network buses / lines Number of capacitor sizes / load levels Line impedance i from bus a to b

2 Reinforcement Learning Reinforcement learning (RL) [20] can be described as a computational approach to learning through interaction with an environment. In a sequential decision task, an agent interacts with an environment, by selecting actions that affect state transitions to optimize some reward function. Formally, at any given time t, an agent perceives its state st and selects an action at . A dynamic system responds by giving the agent some numerical reward r(st ) and changing into state st+1 = δ(st , at ) [21]. The agent’s aim is to find (or to learn) a policy π : S → A, mapping states to actions, that maximizes some long-run measure of reinforcement. π ∗ = argmaxπ V π (s) , ∀s

(1)

where V π (s) is the cumulative reward received from state s using policy π , called value-function. The most common approach to learning value functions is the temporal difference (TD) methods. These methods can learn directly by experience without any explicit model of environment’s dynamics. Furthermore, they update estimates based on previous learned estimates, without waiting for a final outcome. Defining the action-value function Q (st , at ) = r (st ) + V π (δ (st , at )) ,

(2)

it’s possible to set up the update rule of the off-policy RL algorithm Q-learning [20], on its simplest form. Q (st , at ) ← Q (st , at ) + α∆Q (st , at )

(3)

∆Q (st , at ) = U (st+1 , at+1 ) − Q (st , at ) + r (st ) (4)

U (st+1 , at+1 ) = γ max Q (st+1 , at+1 ) at+1

(5)

where γ and α denote, respectively, the learning rate and discount rate of reinforcements along time. In equation (3), the function Q (st , at ) is updated based on its current value, immediate reward r (st ), and the difference between the maximum action-value at the next state (finding and selecting the action at the next state that maximizes it) minus the action-value function in the current time. Differing from supervised learning techniques, the environment is explicitly considered on a trade-off between exploration and exploitation. The agent must learn which actions maximize gains in time, but also how to act to reach this maximization, looking for actions still not selected or regions not considered in a state space. As both directives bring, in specific moments, benefits to problem solutions, the exploration and exploitation modules are usually mixed. Let the -greedy policy, a policy where the parameter  indicates the probability of choosing a random action, and (1 − ) the probability of choosing the action of larger expected long-run value of return. Then, the greedy action a∗t for state st can be obtained according to equations below [22]. a∗t

= argmaxat ∈A(st ) Q (st , at )

π (st , a∗t ) π (st , at ) =

 =1−+ |A(s)|

 , ∀a ∈ A(s) − {a∗t } |A(s)|

3.1 Problem Formulation The capacitor placement problem consists of determining optimal location, number, size, type, and control scheme of capacitor banks, such that minimum yearly cost due to power losses and cost of capacitors are achieved, while operational and power supply quality constraints are respected. Mathematically, this problem can be formulated initially as a combinatorial optimization problem, with search space size given by Θ(Λ + 1)N , where Λ is the number capacitor sizes, Θ is the number of load levels under analysis and, N is the number of network buses. min f (v, z) = fC (v, z) + fL (v, z)

subjected to

(8)

Fundamental for TD methods convergence, the trade-off between exploration and exploitation as well as policy functions π , value functions V , action-value functions Q, and agent/enviroment interactions, are reinforcement learning paradigm elements, and have been used in the development of proposed approach.

g (v m , z m ) = 0

(10)

V min ≤ (vkm ) ≤ V max

(11)

where fC (v, z) = fL (v, z) =

(6) (7)

Θ X N X

m m κf,s C zk Ck

This section is divided into two subsections. Subsection 3.1 presents problem formulation where placement, control scheme and policy search are mathematically summarized. It is presumed in this first approach constant demand modeling per load level under analysis, and balanced three-phase radial distribution feeders. Subsection 3.2 describes methodology emphasizing problem modeling and algorithm.

(12)

m k Θ X m m m m κm L T PL (v , z ) m

(13)

In equations (9), (12) and (13), the objective function f is divided into costs associated to capacitors banks (purchase and installation) fC and costs associated to electrical losses fL (obtained through the cost coefficient κm L , for Θ load levels, and electrical losses PLm ). The variable zkm denotes shunt capacitance existence at node k for load level m. Equations (10) and (11) correspond to load flow and voltage magnitudes constraints, respectively. This last constraint has been considered through the addition of a penalty factor β to the voltage deviation as followings: f (v, z) = fC (v, z) + fL (v, z) + β

3 Proposed Approach

(9)

N,Θ X

φvkm

(14)

k,m

subjected to g (v m , z m ) = 0

(15)

where ( φvkm =

0.01, if V min ≤ vkm ≤ V max 0.5 abs 1 − (vkm )2 , otherwise

(16)

Pointing out policy search, financial constraints like-

wise number (or size) limits to capacitors banks at buses, feeders and/or distribution substations, are not directly included in such object function formulation. Conversely, these constraints are considered in aiming for the most profitable ordination for capacitors placement. In fact, although equation (14) is suited to optimal location search and control scheme search of capacitors banks in distribution system, an optimal capacitor placement policy search is also proposed in this approach. For this purpose, let s be the placement state, a bi-univocal function of z m C m , ∀m m

m−1

z}|{ z}|{ → − m m k ... ]ΘN s = [ ... k ... sm s s ... k k+1 } | k−1 {z

(17)

→ s = h(− s ), such that ∃h−1 , h : NΘN → N

(18)

N

and let a ∈ A be the action of installing a capacitor bank of size and type Ckt,Θ at bus k , ∀k . Hence, let Q a function representing the expected value of allocating each capacitors bank, whereas the placement state s, the optimal placement policy is conveniently summarized through the equation (19) bellow. π ∗ (s) = argmaxa Q (s, a)

At this point, two heuristic rules are described below. 3.2.1 Heuristic I: Immediate Reward Estimation During the first visit of a placement state, it’s specified as a directive of the methodology, the choose of the action associated with the larger expected immediate reward. Common in applications to the problem, this procedure requires extensive object function numerical evaluations. Aiming to reduce the computational burden required, it’s performed an immediate reward estimation for each possible action given a placement state, as followings. Let sj be the state obtained in time j of agents’ iterative learning. Also let aju be the action representing a capacitor bank installation at bus u such that Qm ju = Cjmu zjmu , for Θ load levels. Moreover, consider now vami and vbmi , the initial and final bus voltages at line i for load level m. Then the object function obtained for state sj+1 can be estimated through sensitivity voltage [23] evaluations, related to reactive power injection, as shown below. fj+1 ≈ fj +

κf,s C

+

Θ X

m κm LT

m

(19) + β

N,Θ X

∂PLm ∆Qm u ∂Qm u

m φvk+1

(21)

k,m

3.2 Methodology

where

The proposed methodology is based on modeling an agent objecting to learn and discover the optimal location, placement policy, and control scheme of capacitor banks, by means of try-and-error interactions in an environment. This environment is defined as the electric network where the installation of each capacitor bank is an agent’s action with this environment. Hence, the optimal placement policy is referred to actions, in each network status, which maximize a future reward function obtained until the optimal placement state. Considering different load levels in problem formulation, this optimal placement state includes also the optimal capacitor control scheme. Throughout iterative learning, the optimal placement state search (that is, the state associated to the optimal immediate return) 1 r (st ) ≡ (20) f (v, z)

3.2.2 Heuristic II: Reduction of the Search Space Sensitivity-based analysis is also performed for reducing the search space, limiting the search to the η(%) most sensitive buses in relation to objective function.

is self–managed by the learning technique, sensitivitybased analysis, and the use of potential heuristic rules developed and/or already available for the problem.

3.2.3 Proposed Algorithm Based on TD method Q-learning, the proposed algo-

NL 2 vami − vbmi ∂PLm X = i ∂Qm Zab u i




NL 2 vami − vbmi ∂PLm X = i ∂Qm Zab u i




∂vbmi ∂vami − ∂Qm ∂Qm u u



∂vbmi ∂vami − ∂Qm ∂Qm u u



(22)

(23)

and ( m φvk+1 =

m ≤ V max 0.01, if V min ≤ vk+1
2  m , otherwise 0.5 abs 1 − vk+1

m vk+1 ≈ vkm +

∂vkm ∆Qm u ∂Qm u

(24)

(25)

rithm is outlined below. 1) Read input data (line and bus data) and initialize parameters and variables. 2) Define search space considering the η(%) most sensitive buses with respect to objective function. 3) Start with state representing the uncompensated reactive power status of the electric network. 4) If the placement state is visited for the first time, choose the action associated with the larger expected immediate reward. This procedure can be performed through immediate reward estimation of all possible actions for the placement state through equation (21). Otherwise, choose an action under -greedy policy defined by the equations (6), (7), and (8). 5) Obtain next placement state from current state and current action. States not visited and respective immediate reward (optionally) are stored in memory. 6) Update action-value function using equations (3) and (5). 7) If the immediate reward of next state is smaller than the immediate return of current state under greedy action, return to Step 4. Otherwise, go to Step 8. 8) Update learning rate α and probabilities 
αiter+1 = max 0.95αiter , αf inal (26) iter+1 = max 0.95iter , f inal




(27)

9) If the π policy convergence is characterized, go to Step 10. Otherwise, return to Step 3. X Qiter (sj , aj ) − Qiter−1 (sj , aj ) < Ψ (28) 10) Computation, impression of numerical results and, end of iterative process.

4 Simulation Results 4.1 Case study description The proposed approach for optimization and automatic control of reactive power has been tested in a 13.8 kV, 29-bus Brazilian Central Region real system. The annual load curves were segmented in three load levels (light, intermediate, peak) of demand factors and duration T , specified in Table 1. Fixed and switched capacitor purchase and instal-

Light 0.30 3,102.50

Demand factor T (hours/year)

Intermediate 1.67 4,562.50

Peak 2.00 1,095.00

Table 1. Load levels. lation costs are shown in Table 2 [24]. Simulations considered energy losses costs by the coefficient κL = 0.13380 US$/kWh for the three load level under analysis. Upper and Lower bound voltage magnitudes, according to voltage levels standards in [25], are set up in V min = 0.93 pu and V max = 1.05 pu. The parameter β will have been calibrated to represent Brazilian regulatory penalties. For this preliminary simulations, β was used as a penalty factor to capacitor placements associated to high voltage deviation. Type Fixed Switched

Size (kVAr) 600 1,200 0/600 0/1,200 0/600/1,200

Cost (US$) 3,091 3,909 5,818 6,636 8,455

Table 2. Capacitor costs.

4.2 Result Analysis After several numerical simulations, penalty factor was adjusted in β = 100000. Training parameters used on final outcome are shown in Table 3. αinitial 0.90

αf inal 0.20

γ 0.95

initial 0.30

f inal 0.01

Table 3. Training parameters. The proposed algorithm presented good convergence and adequate robustness to variations of these parameter. It’s recommended initial learning rates values near to unit (typical values are 0.75 to 0.90), and low final learning rates (typical values are 0.1 to 0.3). For initial and final probabilities , typical values are 0.2 to 0.3 and 0.01 to 0.1, respectively. In addition, discount rates γ near to unit (typical values are 0.80 to 0.95), modeling high influences of long-run reinforcements on state values, increase significantly algorithm performance. Table 4 summarizes the optimal capacitor placement solution, control scheme and policy, resulted for the application of proposed methodology in the Brazilian case study described previously. For each step of the capacitor placement policy solution, are indicated

capacitor type, size, bus location, and control scheme, obtained. Step

Bus

Type

0 1 2 3 4 5 6 7 8

29 28 26 24 21 20 18 17

Fixed Fixed Fixed Fixed Fixed Switched Fixed Fixed

Light 1,200 1,200 1,200 1,200 1,200 600 1,200 1,200

kVAr Interm. 1,200 1,200 1,200 1,200 1,200 1,200 1,200 1,200

Peak 1,200 1,200 1,200 1,200 1,200 1,200 1,200 1,200

Table 4. Capacitor placement solution, control scheme and policy, obtained. Table 5 shows results for each capacitor placement policy step in terms of electrical losses PL , capacitor purchase and installation accumulated cost fcac , and 0 object function cost regardless voltage penalty f . Step 0 1 2 3 4 5 6 7 8

Bus 29 28 26 24 21 20 18 17

fcac (US$) 3,909.00 7,818.00 11,727.00 15,636.00 19,545.00 28,000.00 31,909.00 35,818.00

PL (kW) 418.72 395.68 376.22 359.67 345.79 334.18 324.48 316.48 310.02

0

f (US$) 296,418.66 278,353.58 264,086.35 252,932.30 244,571.42 238,575.62 238,318.58 235,916.05 235,274.34

Table 5. Allocation policy results. A comparative between uncompensated network and compensate network is shown in Table 6, considering additionally the maximum and minimum voltage magnitudes vmin , vmax , and savings Sav obtained.

vmin (pu) vmax (pu) PL (kW) 0 f (US$) Sav (US$)

Uncompensated Compensated 0.929 0.953 0.993 1.002 418.72 310.02 296,418.66 235,274.34 61,144.32

Table 6. Reactive compensation effect. As observed in tables above, capacitor installation at this distribution feeder performs meaning electrical

losses reduction and voltage profiles improvement. The solution indicated installation and purchase of seven 1,200 kVAr type fixed and one 0/600/1,200 kVAr type switched-capacitor bank. Total losses without any compensation is found to be 418.72 kW. After compensation, the total losses is 310.02 kW, equivalent 26% reduction. Savings are estimated to be US$ 61,144.32. From policy placement solution, electric utilities can set up a compensation strategy, differentiated in steps, leading to optimal capacitor placement state, minimizing losses, purchase and installations costs. For example, given an annual budget for system improvements of US$ 16,000, policy solution indicates the purchase and installations of four 1,200 kVAr type fixed-capacitors banks (Steps 1 to 4 in Table 5), as the optimal strategy to reactive power compensation. Futhermore, policy placement solution can aid in budgetary assessments for system improvements. Finally, preliminary results of proposed approach applied to a real system suggest effectiveness and robustness in power distribution systems optimization problems. Problem formulation and solution, provide decision support to reactive power compensation planning in large scale electric utilities.

5 Conclusion A new methodology for optimization and automatic control of reactive power in distribution systems is proposed in this work. The capacitor placement policy search was approached to provide decision support to planning compensation in large scale energy companies. Based on reinforcement learning concepts, a formulation to a concomitant search for optimal placement, optimal placement policy, and optimal control scheme of capacitor banks in distribution networks was presented. In fact, the reinforcement learning technique showed effectiveness in landscape combinatorial optimization problems. Sensitivity-based analysis improve the proposed method adding knowledge to the model and speeding up simulations. The designed algorithm was applied in a Brazilian Central Region real system to improve voltage profiles and reduce electric power losses. Preliminary results pointed up good performance and robustness in power distribution systems optimization problems. Future work needs to be carried out with regard to the following issues: nonlinear and unbalanced loads, annualized maintenance costs, and metaheuristic methods hybridism.

Acknowledgment The authors would like to acknowledge the financial, technical and human support provided by the Goias Energy Company (CELG) and the Coordination for the Improvement of University Level Human Resources (CAPES).

References [1] C. Lyra, C. Pissara, C. Cavellucci, A. Mendes, P. M. Franc¸a. Capacitor placement in largesized radial distribution networks, replacement and sizing of capacitor banks in distorted distribution networks by genetic algorithms. In IEEE Proceedings Generation, Transmision & Distribution, 2005, pp. 498–516. [2] A. Y. Chikhani, H. N. Ng, M. M. A. Salama. Classification of capacitor allocation techniques, Vol.15, No.1, 2000. [3] N. M. Neagle and D. R. Samson. Loss reduction from capacitors installed on primary feeders, Vol.75, 1956, pp. 950–959. [4] J. V. Schmill. Optimum size and location of shunt capacitors on distribution feeders, Vol.84, No.9, 1965, pp. 825–832. [5] M. H. Haque. Capacitor placement in radial distribution systems for loss reduction, In IEE Proceedings Generation, Transmission and Distribution, 1999, pp. 501–505. [6] I. E. I.-Samahi, M. M. A. Salama, E. F. ElSaadany. The effect of harmonics on the optimal capacitor placement problem, 2004. [7] F. F. Wu and M. E. Baran. Optimal capacitor placement on radial distribution systems, Vol.4, No.1, 1989, pp. 725–734. [8] F. F. Wu and M. E. Baran. Optimal sizing of capacitor placed on a radial distribution systems, Vol.4, No.1, 1989, pp. 735–743. [9] A. Jafarian, E. F. Fuchs, M. A. S. Masoum, M. Ladjevardi. Fuzzy approach for optimal placement and sizing of capacitor banks in the presence of harmonics, Vol.19, No.2, 2004. [10] H. C. Chin. Optimal shunt capacitor allocation by fuzzy dinamic programing, Vol.35, 1995, pp. 133–139. [11] R. Annaluru, S. Das, and A. Pahwa. Multi-level ant colony algorithm for optimal placement of capacitors in distribution systems, IEEE Press, 2004, pp. 1932–1937.

[12] C. T. Su, J. P. Chiou, C. F. Chang. Ant direction hybrid differential evolution for solving large capacitor placement problems, Vol.19, No.4, 2004. [13] L. P. Lern, C.S. Chang. Application of tabu search strategy in solving nondifferentiable savings function for the calculation of optimum savings due to shunt capacitor installation in a radial distribution system, In IEEE PPower Engineering Society Winter Meeting, 2000, pp. 2323–2328. [14] S. Tsunokawa and H. Mori. Variable neighborhood tabu search for capacitor placement in distribution systems, Vol.3, 2005, pp. 4747–4750. [15] O. T. Tan and N. I. Santoso. Neural-net based real-tme control of capacitor installed on distribution systems, Vol.5, No.1, 1990, pp. 266–272. [16] M. Begovic and B. Milosevic. Capacitor placement for conservative voltage reduction on distribution feeders, Vol.19, No.2, 2004. [17] A. G. Exposito, J. L. M. Ramos, J. R. Santos. A reduced-size genetic algorithm for optimal capacitor placement on distribution feeders, 2004. [18] S. N. Kim, S. K. You, K. H. Kim, S. B. Rhee. Application of esga hybrid approach for voltage profile improvement by capacitor placement, Vol.18, No.4, 2003. [19] R. Romero R. A. Gallego, A. J. Monticelli. Optimal capacitor placement in radial distribution networks, Vol.16, No.4, 2001. [20] A. G. Barto and R. S. Sutton. Reinforcement Learning, The MIT Press, 1998. [21] J. J. Grefenstett,e D. E. Moriarty, A. C. Shultz. Evolutionary algorthms for reinforcement learning, Vol.11, 1999, pp. 241–276. [22] G. Bittencourt and E. Camponogara. Genetic algoritms and reinforcement learning. Booklet: Electrical Engineering’s Courses and Lectures of Federal University of Santa Catarina, Brazil, 2005. (In Portuguese). [23] A. B. K. Sambaqui. Improvement of Voltage Profiles Methodologies in Distribution Systems, PhD thesis, Federal University of Santa Catarina, 2005. (In Portuguese). [24] S. Haffner, F. A. B. Lemos, J. S. Freitas, M. V. D. Freitas. Fixed and switched caacitor banks placement in radial distribution systems for different load levels, 2004. [25] Resolution 505. Stead state voltage levels standards. Technical report, Brazilian Electricity Regulatory Agency, 2001. (In Portuguese).