Using Recurrent Neural Networks for Time Series Forecasting of Electric Demand in Sulaimani.
A THESIS Submitted to the Council of College of Administration & Economics - University of Sulaimani, As Partial Fulfillment for the Requirements of the Master Degree of Sciences in Statistics
By
Ayad Othman Hamdin Supervised by: Assistant Professor
Dr. Nawzad Mohammad Ahmed
2016 (AD)
2716(K)
@pbØ@õò−Œ@ð䆋Ø@ôåïj“Žïq@üi@òìa Šó @ñŠbàò†@ðäbØò Šüm@ðäbåŽïèŠbØói ðäbáŽïÝ"@óÜ@biòŠbØ@ñòŒìì@!"aí‚@üi@ @ ò‹Žïn)ubà@ðØóîóàbä@ n)ubà@ðØóîóàbä@ @óÜ@ÚŽï’ói@Ûòì@ðäbáŽïÝ"@ñüÙäaŒ@M@ õŠŠíib÷@ì@ñ ‹Žï ŠbØ@ðvŽïÜüØ@ðäóàí−ó÷@ói@òìa‹Ø@•óÙ“Žïq Šbàb÷@!"aŒ@óÜ@‹Žïn)ubà@õóÝq@ðäbåŽïén"ò†òì@ðäbØóïn)îìa‡Žïq@ q@ @
çóîýóÜ@ çóîýóÜ@
æ@çbárÈ@†bîc@ æ@çbárÈ@†bîc@ @ @
@!’ŠóqŠó"ói@ Šò†ò‡îŠbî@ñŠü)ïÐû‹q@
‡¼c@‡á«@†aŒíä@N†@ @ 2016(AD)
2716(K)
@ôÝÈ@kÝ%ÝÜ@óïåà'Üa@Þþ*Übi@üjånÝÜ@òŠ‹Ùn¾a@óïj—ÉÜa@pbÙj“Üa@ãa‡ƒng @ @óïäbáïÝ*Üa@À@öbi‹éÙÜa @ @ óà‡Õà@óÜbŠ@@ óïäbáïÝ*Üa@óÉàbu@–@†b—nÔfiaì@òŠa†fia@óïÝØ@5ݪ@¶g@ öb—yfia@À@ãíÝÈ@;n*ubà@óuŠ†@Þïä@pbjÝ%nà@æà@ö'u@ðèì@ @ @@
ÞjÔ@æà@
æ@çbárÈ@†bîc@@ @ @ @Óa‹’hi@ ‡Èb*¾a@ˆbnÿa@
‡¼c@‡á«@†aŒíä@N†@ @
2016(AD)
2716(K)
@ @@@@@@@@@
@ @{@ŞpbŽuŽŠŽ†@Žâ ÝđÉ Üa@aížmìc@Žæîđ‰ÜaŽì@ĆâÙĆåđà@aížåŽà e@Žæîđ‰Üa@žêÝÜa@"Ê ÐĆ‹Žî@} 11@Zó܆b(a@òŠí*@
{@bćàaŽí Ô@ ÚđÜٰŽˆ@ŽæĆïŽi Žçb ØŽì aìž‹žn ÕŽî Ćâ ÜŽì aíÐ"‹Ć1žî Ćâ Ü aíÕ ÑĆä c aŽˆ"g@Žæîđ‰ÜaŽì@}@@@@ @67@ZçbÔ‹ÑÜa@òŠí*
DEDICATION
THIS THESE DEDICATE TO MY WIFE (ZHILYA) AND MY SON (ARYA) MY PARENT, BROTHERS AND SISTERS. MY SUPERVISOR ASST.PROF DR.NAWZAD M.AHMED. TO ALL MY TEACHERS TO ALL PERSON WHO WISH ME TO SUCCESS.
I
ACKNOLEDGMENTS
All praise to Allah for the strength and his blessing in completing this dissertation. First and foremost, it is my pleasure to thank those people who made this dissertation possible. It is difficult to overstate my gratitude to my Supervisor, Asst. Prof Dr.NawzadM.Ahmed who, with his inspiration, enthusiasm, insightful comments and great efforts explained things clearly which helped to make this dissertation possible. My sincere thanks also go to the dean of the collage (Asst. Prof Dr. Kawa M. Jamal), (Asst. Prof Dr. Muhammad. M faqe) who is the head of statistics department and all the other lectures from the Department of Statistics, who taught me statistics, for their support and encouragement. My thanks also to my teachers who inspired me through my MS.c study, especially Prof. Dr. Monem Aziz, Asst. Prof. Dr. Shawnim Abdulkader, Asst. Prof. Dr. Nzar Abdulkader, Asst. Prof. Dr. Samira M. Salih and Dr. Sozan S. Haidar. I would also like to express my gratitude to M. Farhad A. Ahmed and M. Shaho. M Wstabdullah from the Department of statistics at the University of Sulaymaniyah, who support me to use software and preparing my dissertation. Again, my sincere thanks goes to the council of Sulamani University, the council of School of Administration and economic, my entire family and all my friends, for providing me sound advice. Finally, I would like to thank the library of College of Economic and Administration / Sulaimani University for their help during my work.
II
Abstract Electricity is counted a one of the most important energy sources in the world. It has played a main role in developing several sectors, such as: Economy, industry, electronics… etc. In this study two types of electricity variables have been used, the first was the demand on the power, and the second was the consumption or load on the power in Sulaimani city. The main goal of the study was to construct an analytic model of the recurrent neural network (RNN) for both variables. This model has a great ability in detecting the complex patterns for the data of a time series, which is most suitable for the data concerning electric energy. This model is also more sensitive and reliable than the other recurrent neutral network (RNN), so it can deal with more complex data that might be chaotic, seismic….etc. this model can also deal with nonlinear data which are mostly found in time series, and it deals with them differently compared to the other models. This research determined and defined the best model of RNN for electricity demand and consumption to be run in two levels. The first level is to predict the complexity of the suggested model (1-5-10-1) with the error function as (MSE, AIC, and R2). The second level uses the suggested model to forecast the demand on electricity and the value of each unit. Another result of this study is finding the right algorithm that can deal with such complex data. The algorithm (Levenberg-Marquardt) was found to be the most reliable and has the most optimum time to give accurate and reliable results in this study.
III
TABLE OF CONTENTS Title
Page
ACKNOWLEDMENTS
II
ABSTRACT
III
TABLE OF CONTENTS
IV
LIST OF ABBREVATION
X
Chapter One: Introduction, Literature review and Aim of thesis 1-1
Introduction
1
1-2
Aim of thesis
4
1-3
Literature Review
4
1-4
Layout of thesis
11
Chapter Two: Theoretical Part 2-1
Introduction
12
2-2
Artificial Neural Networks (ANNs)
12
2-3
There are some types of artificial neural networks
12
2-4
Architecture of ANNs
15
2-4-1
Input layer
16
2-4-2
Hidden layer
17
2-4-3
Output layer
17
2-5
Formally Recurrent Neural Networks (RNNs)
17
2-6
Recurrent Neural Networks (RNNs)
19
2-7
Activation function
20
2-8
some types of activation function
21
2-8-1
Hard Limit activation Function
21
2-8-2
Linear activation Function
21
2-8-3
Log-Sigmoid activation Function
22
IV
2-8-4
Hyperbolic activation Function
23
2-8-5
Soft-Max activation Function
23
Neural Network Training
24
2-9-1
Supervised Training
25
2-9-2
Unsupervised Training
25
2-9-3
Reinforcement Training/ Neurodynamic Programming
26
2-10
Weights
26
2-11
Bias
27
2-12
Artificial Neurons
28
2-13
Difference between RNNs and FFNNs
29
2-14
Neural Networks with Algorithms
30
2-15
Levenberg-Marquardt algorithms (LMA)
30
2-16
Derivation of Levenberg-Marquardt algorithm
31
2-16-1
Steepest Descent algorithm
33
2-16-2
Newton’s Method
34
2-16-3
Gaussian-Newton Algorithm
37
2-16-4
Levenberg-Marquardt Algorithm Rule
39
Algorithm Enforcement
41
2-17-1
Calculation of Jacobian Matrix
41
2-17-2
Training Process Design
48
some types of measure important for choose the best network
50
2-18-1
Akaike Information Criterion (AIC)
50
2-18-2
Mean Square Error (MSE)
51
2-18-3
Coefficient of determination (R2)
52
Time series analysis and prediction
53
Time-series
54
Time series Analysis
54
2-9
2-17
2-18
2-19 2-19-1 2-19-1-1
V
Chapter Two: Theoretical Part 2-19-1-2
Problems with time series Analysis
55
2-19-1-3
Time series prediction
55
Methods for Time series prediction
56
2-19-2-1
linear models
56
2-19-2-2
Moving Average Models (MA)
56
2-19-2-3
Autoregressive Models (AR)
57
2-19-2-4
Mixed Autoregressive and Moving Average Models (ARMA)
58
2-19-2-5
Non-linear Models
58
2-20
Non-linear autoregressive moving average model (NARMA)
59
2-21
Recurrent Neural Networks versus Feedforward Models
60
2-22
Forecasting versus Prediction
62
2-19-2
Chapter Three: Application Part 3-1
Introduction
63
3-2
Recurrent Neural Network Design
63
3-2-1
Data Description
63
3-2-2
The Application Steps of Recurrent neural Networks
65
3-2-3
Results: Prediction and Forecasting steps
78
Results and Discussions
87
3-3
Chapter Four: Conclusions and Recommendations 4-1
Conclusions
90
4-2
Recommendations
92
References
93
Appendices
102 ó–þ©a@ ón‚íq@
VI
LIST OF TABLE Table No.
Title
Page
2-1
Represent the characteristics of sample activation function
24
2-2
Summarize the update rule for various algorithm (Specifications of Different Algorithm)
40
3-1
The sample of data used to application
64
3-2
Represent the best architecture of (RNN)
67
3-3
represents finding the best architecture of RNN model
70
Finding the best architecture of RNN model for data under 3-4 3-5
71
consideration. Represent finding the best activation function for the best
76
architecture network [1-5-10-1]. Represents the result of applying the (RNN) model (1-5-10-1) for 3-6
(141Obs) in validation set. (Prediction)
79
Represents the result of applying the (RNN) model (1-5-10-1) for 3-7
(60) observation after (940) observation. (forecasting)
84
Represents the result of forecasting for two months after (940) 3-8
85
observation. Represents the Differences between actual data and forecasting
3-9
for two months. (D= Actual data – Forecast data)
VII
86
LIST OF FIGURES Figure 2-1
Title Represented the architecture of ANNs.
Page 16
Recurrent neural network where connections between units form 2-2
18
a directed.
2-3
Simple Recurrent Neural Network
20
2-4
Represent the hard-limit function
21
2-5
Represent the linear function (purelin)
22
2-6
Represent the log-sigmoid function
22
2-7
Represent the tan-hyperbolic function
23
2-8
Represent the Soft-max function
23
2-9
Represents the weight values corresponding to the strength of synaptic connections
27
2-10
The network with Bias
28
2-11
Biological and artificial neuron design
29
2-12
Shown the differencing between RNNs and FFNNs
30
2-13
Represented connection between neurons of the network
42
2-14
Represent training using Levenberg-Marquardt algorithm
49
3-1
Represent the best architecture RNN model (1-5-10-1).
67
3-2
Show the training state.
68
3-3
Represent the training performance.
69
VIII
Shown that plot regression of (training, testing, validation and all 3-4
77
data).
3-5
Represent the error histogram in training.
78
3-6
Represents the difference between Actual data and prediction.
81
3-7
Represents the weight distribution.
83
3-8
Represents the actual data and forecast data for two months.
86
IX
List of Abbreviation: Abbreviation
Full Name
ANN
Artificial Neural Network
AARTRL
Adaptive Amplitude Real Time Recurrent Learning
AEST
Absolute Exponential Stability
AIC
Akaike Information Criterion
AR
Auto regressive
ARMA
Auto regressive Moving Average
ARMCRN
Auto-Regressive Multi Context Recurrent Neural Network
BRNN
Bidirectional Recurrent Neural Network
BM
Brownian Motion
EA
Evolutionary Algorithm
EBP
Error Back-Propagation
ELF
Electricity Load Forecasting
FFNNs
Feed Forward Neural Networks
GA
Genetic Algorithm
GES
Global Exponential Stability
X
GNA
Gauss–Newton Algorithm
GRNN
General Regression Neural Network
LMA
Levenberg-Marquardt Algorithm
LSTM
Long, Short-Term Memory
MA
Moving Average
MLP
Multilayer Perceptron
MSD
Mean Square Deviation
MSE
Mean Square Error
NARMA
Non-linear Autoregressive Moving Average
NN
Neural Network
RBP
Recurrent Back-Propagation
RL
Reinforcement Learning
RNNs
Recurrent Neural Networks
XI
Chapter One: Introduction, Literature review and Aim of thesis
1-1 Introduction Artificial Neural Networks are comparatively crude electronic models based on the neural network structure of the human brain. The human brain essentially learns from experience. It is natural proof that some issues are beyond the domain of current computers are indeed solvable by small energy efficient packages. This human brain modeling also promises less technical way to develop machine solutions. This new approach to computing also provides more nimble degradation during system overload than its more traditional counterparts. [9] The name of artificial neural networks (ANN) indicates computational networks that attempt to simulate, in a great manner, the networks of neurons of the biological (human or animal) central nervous system. This simulation is a great (neuron-by-neuron, cell-by-cell) simulation. It borrows from the neurophysiologic knowledge of biological neurons and of networks of such as biological neurons. As an outcome, it differs from conventional (digital or analog) computing machines that serve to replace, develop or speed-up human brain computation without regard to organization of the computing elements and of their networking. Still, it can be accentuated that the afforded of simulation by neural networks is very great [11]. Furthermore, Recurrent Neural Networks (RNNs) are sensitive type of artificial neural network models that are well suitable for pattern classification functions whose inputs and outputs are sequences. The importance of developing methods for mapping sequences to sequences is exemplified by functions such as (speech recognition, speech synthesis, named-entity recognition, language modeling, and machine translation). An RNN represents a 1
Chapter One: Introduction, Literature review and Aim of thesis
sequence with a high-dimensional vector (called the hidden state) of a fixed dimensionality that incorporates new observations using an intricate nonlinear function. RNNs are very expressive and can implement arbitrary memory-bounded computation, and as a result, they can likely be configured to achieve nontrivial performance on difficult sequence functions. However, RNNs have turned out to be difficult to train, especially on problems with complicated long-range temporal structure – precisely the setting where RNNs ought to be most useful. Since their potential has not been realized, methods that address the difficulty of training RNNs are of great importance.[7] Recurrent neural networks (RNNs) are subclass ANNs connectionist models that capture the dynamics of sequences via cycles in the network of nodes. Unlike standard feed forward neural networks, recurrent neural networks keep a state that can represent information from an arbitrarily long context window. Although recurrent neural networks have conventionally been difficult to train, and often contain millions of parameters, recent advances in network architectures, optimization techniques, and parallel computation have enabled successful large-scale learning with them. In recent years, systems based on long short-term memory (LSTM) and bidirectional (BRNN) architectures have demonstrated ground-breaking performance on tasks as varied as image captioning, language translation, and handwriting recognition. [33] Recurrent neural networks have been an interesting and important part of neural network research during the 1990's. They have already been applied to wide variety of problems involving time sequences of events and ordered data such as characters in words. Novel current uses range from motion
2
Chapter One: Introduction, Literature review and Aim of thesis
detection and music synthesis to financial forecasting .Challenges in this subfield of artificial neural network research and development by sharing these perspectives. Learning is a critical issue and one of the primary advantages of neural networks. The added complexity of learning in recurrent networks has given rise to a variety of techniques and associated research projects. A goal is to design better algorithms that are both computationally efficient and simple to implement. The next decade should produce significant improvements in theory and design of recurrent neural networks, as well as many more applications for the creative solution of important practical problems. The widespread application of recurrent neural networks should foster more interest in research and development and raise further theoretical and design questions, Recurrent neural networks (RNNs) are connectionist models with the ability to pass selectively information across sequence steps, while processing sequential data one element at a time. Thus they can model input and/or output consisting of sequences of elements that are not independent. Further, recurrent neural networks can simultaneously model sequential and time dependencies on multiple scales. [17]
3
Chapter One: Introduction, Literature review and Aim of thesis
1-2 Aims of the thesis The main goal for using this type of neural network that named by Recurrent Neural Network (RNN) is to use to recognize the patterns included in this kind of time series as a chaotic, seismic, and Brownian motion because not all types of artificial neural network can make these properties clear, also making forecasting with (RNN) in demand of electric power is the first and secondly to know what happen in the feature about the behaviors of the time series under consideration, moreover make an RNN model to determine these behaviors in time series under consideration and making it more generalize network model.
1-3 Literature Review: Generally the artificial neural network is an art in addition to that it is a more default types of models especially in time series analysis. According to the historical development of Recurrent Neural Networks (RNNs), several cases and previous studies can be argued. RNN is a kind of artificial neural networks, which is a hard kind of neural network model. Christofer Brax (2000) claims that RNNs for time series prediction compared with time delayed feed forward networks, feed forward neural networks and linear regression models on a prediction task. The results show that the RNN is not better than the other evaluated algorithm. In fact, the time delayed feed forward showed gives the best prediction. [32] In addition 2001 during a study C.Lee, Steve Lawrence and Ah Chung Tsoi used noisy time series prediction by using RNNs and grammatical inference is financial forecasting. This is an example of a signal processing
4
Chapter One: Introduction, Literature review and Aim of thesis
problem which is challenging due to small sample size, high noise, nonstationary, and non-linearity. The method proposed uses conversion into a symbolic representation with a self-organizing map, and grammatical inference with RNN. [33] Moreover, in 2001 FELIX GERS stated that long and short term memories in RNNs. For example, RNNs were thought to be theoretically fascinating for a long time. This is unlike standard feed forward recurrent neural networks (RNNs) that can deal with arbitrary input sequences instead of static input data only. This combined with the ability to memorize relevant events over time. This also makes RNNs in principal more powerful than standard FFNNs. [39] Also, P.Tino, C.Schittenkopf and G.Dorffner in (2001) investigated that “Financial Volatility Trading Using Recurrent Neural Networks”. The main predictive models studied are recurrent neural networks (RNNs). In this study, the applications have been studied in isolation. However, due to special character of daily financial time-series, it is difficult to make full use of RNN representational power. RNNs are either tending to overestimate noisy data, or behave like finite memory sources with shallow memory; they hardly beat classical fixed order Markov model. [50] In 2001 A.Blanco, M.Delgado, M.C.Pegalajar applied real coded genetic algorithm for training in (RNNs). This paper is a comparison between FFNNs and RNNs about algorithm for training networks. This presents also a Real-Coded Genetic Algorithm that uses the appropriate operators for this encoding type to train RNNs. [28]
5
Chapter One: Introduction, Literature review and Aim of thesis
In addition, S.L.Goh and D.P.Mandic (2003) state that algorithm for training in RNNs is adaptive amplitude real time recurrent learning (AARTRL). This type of algorithm for fully connected with RNNs which is employed as nonlinear adaptive filer. [53] Additionally, in 2004 CARRIE KNERR, B.S represents time series prediction using neural networks, which involves the investigation of the effect of prior knowledge embedded in an artificial fully connected RNNs for the prediction of nonlinear time series, algorithm for training used back propagation. This study has compared two network architectures by using time series such as square waves to find best models. [34] Also in the 2004 worked S.Miyoshi, H.F.Yanai and M.Okada research about associative memory by using RNNs with delay elements, the synapses of real neural systems seem to have delayed. In this paper, the researchers worked in re-derive the Yanai-Kim theory, which involves macro dynamical equations for the dynamics of the network with serial delay element. [47] In addition to the previous papers (2004), Erik, Hulthen., Studied two kinds of neural networks for continuous time series (RNN) and (FFNN), they found that the last was suffered from lack of short memory, in addition back propagation only tunes the weights of the networks, and doesn’t generate an optimal design, therefore, the researcher trained RNNs with an evolutionary algorithm (EA) and found that the RNN with (EA) was hard trained. [37] In 2007, Tarik Rashid, B.Q.Huang, M-T.Kechadi and B.Gleeson. This paper presents an auto-regressive network called the Auto-Regressive Multi Context Recurrent Neural Network (ARMCRN) for Electricity Load
6
Chapter One: Introduction, Literature review and Aim of thesis
Forecasting (ELF), which forecasts the daily peak load for two large power plant systems. The autoregressive network is a combination of both recurrent neural networks and non-recurrent neural networks. Weather component variables are the key elements in forecasting because any change in these variables affects the demand of energy load. So the (ARMCRN) is used to complete the relationship between past and future exogenous and endogenous variables. [54] Herrn, Anton and Maximilian, Schafer (2008) contacted a study about learning of (RNN). They found that there are some kinds of learning recurrent neural network, in this study they focused on Reinforcement learning, according to this study, there is a connection between (RNNs) and Reinforcement learning (RL) techniques. Hence, instead of focusing on algorithms, then neural network architecture are put in the foreground as a first step towards reinforcement learning, it is shown that RNN can well map and reconstruct (partially observable) Markov decision processes. [40] Also,J.Xu, Y.Y.Cao, Y.Sun and J.Tang, (2008), they concentrated on proposed (RNNs) with a generalized activation function, in this proposed model, every component of the neuron’s activation function belongs to a convex hull which is bounded by two odd symmetric piecewise linear functions that are convex or concave over the real space, and the absolute exponential stability (AEST) of the recurrent neural network with a generalize activation function class. This study is divided into three types. The first step is to demonstrate the global exponential stability (GES), the second step transforms the RNNs under every vertex activation function in neural networks under an array of saturated linear activation function and the
7
Chapter One: Introduction, Literature review and Aim of thesis
last step is to study both the existence of equilibrium point and the GES of the RNN under linear activation function. [41] Amin-Naseri & RostamiTabar (2008) proposed the use of RNNs. The network consists of four layers; an input layer, a hidden layer, a context layer and an output layer. They used real data sets of 30 types of spare parts from Arak petrochemical company in Iran and three performance measures: Percentage Best (PB), Adjusted Mean Absolute Percentage Error (A-MAPE) and Mean Absolute Scaled Error (MASE). [29] To enrich the previous arguments, Y.Zhao, H.Gao, J.Lam and K.Chen, (2009) investigated the stability analysis for a discrete time in (RNNs) with stochastic time delays as a random variable's drawn from some probability distributions. [56] W.Lin and G.Chen, (2009) also stated that the large memory capacity in chaotic artificial neural networks. The researcher’s analyzed model chaotic with monotonic activation function such as sigmoidal function, this subject shows that any finite-dimensional neural network model with periodic activation functions and properly selected parameters. This has much more abundant chaotic dynamics that truly determine the model’s memory capacity and pattern-retrieval ability. [55] Aymen Chaouachi and Ken Nagasaka (2010), used four kinds of Neural Networks (RNN, MLP, RBF, and NNE) to develop and apply one hour a head forecasting of wind speed and 10km rated wind turbine power of wind generation, dependent on comparing the results, they found that (RBF) performs much better than (RNN) and (RNN) which has achieved the lowest forecasting accuracy. [30]
8
Chapter One: Introduction, Literature review and Aim of thesis
C.Feng, R.Plamondon, and C.O’Reilly, equilibrium (2010) view that Necessary and Sufficient under Condition’s for RNNs Model with time delays to Generate Oscillations, in this paper, the existence of oscillations for a class of RNNs with time delays between neural interconnection was investigated, by using fixed point theory and lyapunov functional. They proved that a recurrent neural network might have a unique equilibrium point which is unstable. This particular type of instability, combined with the boundedness of the solutions of the system. This will force the network to generate a permanent oscillation. [35] James, Martens and Ilya,Sutskever, (2011) investigated RNNs with Hessian-Free Optimization. In this work, they resolved the long-outstanding problem of how to train recurrent neural networks (RNNs) on complex and difficult sequence modeling problems which may contain long-term data dependencies. Utilizing recent advances in the Hessian-free optimization approach (Martens, 2010), together with a novel damping scheme, they successfully trained RNNs on two sets of challenging problems. First, a collection of pathological synthetic and Secondly, on three natural and highly complex real world sequence datasets where they found that the method significantly outperformed the previous state-of-the art method for training neural sequence models. [46] A.Y.Alians, E.N.Sanchez, A.G.Lokianov, and M.A.Perez, (2011), state that working estimation state of real-time RNN. A nonlinear discrete-time neural observer for discrete-time unknown nonlinear systems in presence of external disturbances and parameter uncertainties is presented. It is based on a discrete-time recurrent high-order neural network, which trained with an
9
Chapter One: Introduction, Literature review and Aim of thesis
extended Kalman-filter based on algorithm. This brief includes the stability proof based on the Lyapunov approach. The applicability of the proposed scheme is illustrated by real-time implementation for a three phase induction motor. [57] Mohammed Y.ElShar and Mohammed Anwar Rahman, (2012) claim that forecasting electricity demand by using dynamic ANN, in this work electricity demand forecast plays an important role in the energy section, in the future it is also and crucial for economic development in a country. In addition Electricity usage is a rapidly grown phenomenon in the developing regions, Layer recurrent neural network (LRNN) is a dynamic neural network uses feedback and time delay element, the output is based on the current input and the pervious input and output. This paper mainly considers dynamic neural network model that implements LRNN principles to forecast household Electricity demand. [48] Manuel O.Mellado, et al., 2013 claim that the number of hidden neurons in RNN estimation, genetic algorithms is alternatives for optimizing the performance of RNNs by searching for the best number of hidden neurons. Genetic algorithms have been used for heuristics. Three architectures of RNNs were also used to measure the performance with spoken Spanish digits. 13 were the number of hidden neurons was used for a Jordan network to give the best performance. This number let optimize resources in hardware implantation. [49] Karol, Kuna, (2014) compares existing methods for predicting time series in real time using neural networks. This study mainly concentrates on RNNs and online learning algorithms, such as Real-Time Recurrent
10
Chapter One: Introduction, Literature review and Aim of thesis
Learning and truncated Back propagation through Time. In addition to the standard Elman’s RNNs architecture, Clockwork-RNN is examined. Methods are compared in terms of prediction accuracy and computation time, which is critical in real-time applications. Another part of this study is to make experimental implementations of the tested models and working applications in robotics and network traffic monitoring. [44]
1-4 Layout of thesis: This thesis organized in four chapters as follows: chapter one consists of an introduction to Artificial Neural Network, the aim of the study, literature review is about recurrent neural networks and layout of thesis. Chapter two provides a detailed description about (ANN) and (RNN) with algorithm using in this thesis and the importance of activation function, Weight and Bias in Recurrent Neural Networks, Prediction and structure also it reviews the component and introduces some phrases related to (RNN). Chapter three presents the approaches are used for creating the (RNN) and show the results about prediction and forecasting. Finally, chapter four shows conclusions and recommendations.
11
Chapter Two: Theoretical Part
2-1: Introduction: In this chapter, we focus on the important type of artificial neural network
that
is
Recurrent
Neural
Network
(RNN),
and
study
backpropagation algorithm and training. In this thesis the best algorithm used (Levenberg-Marquardt) for training and it shows how to use the (RNN) for time series prediction and forecasting.
2-2 Artificial Neural Networks (ANNs) Artificial Neural Network is an information processing system, which is inspired by the models of biological neural network. It is an adaptive system that changes its structure or internal information that flows through the network during the training time. In terms of definition Artificial Neural Network is Computer simulation of a "brain like" system of interconnected processing units. [51]
2-3 There are some types of artificial neural networks [24]: 1- The first type of ANNs is Feed-forward Artificial Neural Networks which is called Feed-Forward artificial neural network and has only one condition: information must flow from input to output in only one trend with no (back-loops). There are no limitations on number of layers, type of activation function used in individual artificial neuron or number of connections between individual artificial neurons. The simplest networks named by feed-forward artificial neural network is a single perceptron that is only capable of learning linear separable problems.
12
Chapter Two: Theoretical Part
2- The second type of ANNs is Recurrent Artificial Neural Networks which is called recurrent artificial neural network. It is similar to FFNN with no limitations regarding back loops. In these cases information is no longer transmitted only in one direction but it is also transmitted backwards. This creates an internal state (internal memory) of the network which allows it to exhibit dynamic temporal behavior. Recurrent artificial neural networks can use their internal memory to process any sequence arbitrary of inputs. 3- The third type of ANNs is Elman and Jordan Artificial Neural Networks which is named by (Elman network), also referred as simple recurrent network is special case of recurrent artificial neural networks. It differs from traditional two-layer networks in that the first layer has a recurrent connection. It is a simple three-layer artificial neural network that has back-loop from (hidden layer) to input layer through so called context. This type of artificial neural network has internal memory that allowing it to both detect and generate time-varying patterns. 4- The forth type of ANNs is Hopfield Artificial Neural Network Which is named by (Hopfield network), is a type of recurrent artificial neural network that is used to store one or more stable target vectors. These stable vectors can be viewed as memories that the network recalls when provided with similar vectors that act as a cue to the network memory. These binary units only take two different values for their states that are determined by whether or not the units' input exceeds their threshold. Binary units can take either values of 1 or -1, or values of 1 or 0.
13
Chapter Two: Theoretical Part
5- The fifth type of ANNs is Long Short Term Memory Which is named by (Long Short Term Memory).Is one type of the recurrent artificial neural networks topologies. In contrast with basic recurrent artificial neural networks it can learn from its experience to process, classify and predict time series with very long time lags of unknown size between important events. This manufactures Long Short Term Memory to outperform other recurrent artificial neural networks, Hidden Markov Models and other sequence learning methods. 6- The sixth type of ANNs is Bi-directional Artificial Neural Networks (Bi-ANN) which is named by (Bi-directional artificial neural networks).Are designed to predict complex time series. They consist of two individual interconnected artificial neural (sub) networks that performs
direct
and
inverse
(bidirectional)
transformation.
Interconnection of artificial neural sub networks is done through two dynamic artificial neurons that are capable of remembering their internal states. This type of interconnection between future and past values of the processed signals increase time series prediction capabilities. As such these artificial neural networks not only predict future values of input data but also past values. 7- The seventh type of ANNs is Self-Organizing Map (SOM) which is named by (Self-organizing map). It is an artificial neural network that is related to feed-forward networks but it needs to know that this type of architecture is fundamentally different in arrangement of neurons and motivation. 8- The eighth type of ANNs is Stochastic Artificial Neural Network which is named by (Stochastic artificial neural networks). It is a type
14
Chapter Two: Theoretical Part
of an artificial intelligence tool. It can be built by introducing random variations into the network, either by giving the network's neurons stochastic transfer functions, or by giving them stochastic weights. This makes them useful tools for optimization problems, since the random fluctuations help it escape from local minima. Stochastic neural networks that are built by using stochastic transfer functions are often called Boltzmann machine. 9- The final type of ANNs is a Physical Artificial Neural Network which is named by (Physical Network), most of the artificial neural networks are software-based but it does not exclude the possibility to create them with physical elements which base on adjustable electrical current resistance materials. History of physical artificial neural networks goes back in 1960’s when first physical artificial neural networks were created with memory transistors called memistores. Memistors emulate synapses of artificial neurons. Although these artificial neural networks were commercialized they did not last for long due to their incapability for scalability. After this attempt several others followed such as attempt to create physical artificial neural network based on nanotechnology or phase change material.
2-4 Architecture of ANNs Artificial neural network (ANN) is a machine learning approach that models human brain and consists of a number of artificial neurons. Neuron in ANNs tends to have fewer connections than biological neurons. Each neuron in ANN receives a number of inputs. An activation function is
15
Chapter Two: Theoretical Part
applied to these inputs which results output value of the neuron. Knowledge about the learning earning task is given in the form of examples called training examples. Architecture of ANNs consists of (Input layer, Hidden layer and Output layer). [51]
Figure (2--1)) represented the architecture of ANNs [51]
2-4-11 Input Layer The input layer ayer is a layer which communicates with the external environment. Input layer presents a pattern to neural network. Once a pattern is presented to the input layer, the output layer will produce another pattern. It also represents the condition for which purpose we are training 51] the neural network. [51
16
Chapter Two: Theoretical Part
2-4-2 Hidden Layer The hidden layer of the neural network is the intermediate layer between input and output layer. Activation function applies on hidden layer if it is available. Hidden layer consists hidden nodes. Hidden nodes or hidden neurons are the neurons that are neither in the input layer nor the output layer. [51]
2-4-3 Output Layer The output layer of the neural network is what actually presents a pattern to the external environment. The number of output neurons should be directly related to the type of the work that the neural network is to perform.[51]
2-5 Formally Recurrent Neural Networks (RNNs): According to (Rumelhart et al, 1986), The Recurrent neural networks formally define the standard which forms the focus of the work, Given a sequence of input the nets ( , , ,…….., ), the network computes a
sequence of hidden state ( , ,….., ) , and a sequence of prediction or estimation ( , ,……., ), by iterating the equations:[46]
= + + ……………… (2-1)
= ( ) ………………………… (2-2) = + …………………...... (2-3)
= ( ) …………………………...... (2-4)
17
Chapter Two: Theoretical Part
Where:
: is the weight matrices between input layer and hidden layer.
: is the weight matrices between hidden layer and output layer.
: is the matrix of recurrent weights between the hidden layer and itself at adjacent time steps..
, : are the activation functions. : Bias of hidden layer layer.
: Bias of output layer.
Figure (2-2)) recurrent neural network where connections between units form a directed. [46]
18
Chapter Two: Theoretical Part
2-6 Recurrent Artificial Neural Networks (RANNs): Recurrent neural networks are feed forward neural networks augmented by the inclusion of edges that span adjacent time steps, introducing a notion of time to the model. Like feed forward networks, RNNs may not have cycles among conventional edges. However, edges that connect adjacent time steps, called recurrent edges, may form cycles including cycles of length one that are self-connections from a node to itself across time. At time (t), nodes with recurrent edges receive input from the current data point
( ) and also from hidden node values ( ) in the network's previous state. The output (t) at each time (t) is calculated given the hidden node values
( ) at time t. Input ( ) at time (t -1) can influence the output (t) at time
(t) and later by way of the recurrent connections. Two equations specify all calculations necessary for computation at each time step on the forward pass in a simple recurrent neural network. [45] ( ) = ( ( ) + ( ) + )…………….. (2-5)
(t)= ( ( ) + )……………………………… (2-6) Where:
( ) : represent the Input of data.
( ) : Hidden node values at time (t).
( ) : Hidden node values at time (t-1) in the network's previous state.
: Weight between input and hidden layer.
19
Chapter Two: Theoretical Part
: Weight between Hidden layers. : Bias of hidden layer.
, : Activation functions between layers.
: Weight between Hidden layer and Output. : Bias of Output.
(t) : Output of network.
Figure (2-3): Simple Recurrent Neural Network [45]
2-7 Activation Function Most neural networks pass the output of their layers through activation function. These activation functions scale the output of the neural network into proper ranges. The activation value is fed over synaptic connections to one or more other units. It is sometime called a “Transfer’’ and activation function with a bounded range is called “squashing” function and which actually gives the power to the neural network to handle non-linearities. The
20
Chapter Two: Theoretical Part
choice of activation function may strongly influence complexity and performance of neural networks. [5]
2-8 some types of activation function There are some types of activation function using in artificial neural network such as:
2-8-1 Hard Limit Activation Function The hard-limit limit function shown above limits the output of the neuron to either 0, if the input argument net is less than 0, or 1, if net is greater than or equal to 0. [25]
$1 , & ! ' 0* ( !!) " # ……….. (2-7) 0, & ! ) 0
Figure (2 (2-4) Represent the hard-limit function. [25]
2-8-2 Linear Activation ctivation Function The linear function calculates the neuron’s output by simply returning the value passed to it, means. [25] y=x…………. (2-8)
21
Chapter Two: Theoretical Part
Figure (2-55) Represent the linear function (purelin). [25]
2-8-3 Log-Sigmoid Sigmoid Activation Function The most common sigmoid transfer function, so called logistic function is most commonly used through a number of alternative transfer functions takes the from:
=
+, -.
……… (2-9)
Where the output of log-sigmoid function between [0 and 1]. As shown figure (2-6). [25]
Figure (2 (2-6) Represent the log-sigmoid function. [25]
22
Chapter Two: Theoretical Part
2-8-4 Hyperbolic A Activation Function Is another kind of sigmoid function, commonly used in network processing. This sigmoid function produce output in the range of [-1 [ and +1] as shown in figure (2--7). [25] =
, . , -. , . +, -.
………….. (2-10)
hyperbolic function. [25] Figure (2-77) Represent the tan-hyperbolic
2-8-5 Soft-Max Activation ctivation Function Is a generalization of the logistic function that "squashes" the function between range [0, +1] is given by by. [25] Yj =∑1
. , 0
234 ,
.1
……………. (2-11) for j=1,…,I
Figure (2 (2-8) Represent the Soft-max function. [25]
23
Chapter Two: Theoretical Part
Table (2-1) Represent the characteristics of sample activation function. [21]
Activation Function
Function
Linear activation
activation function
Y=
Soft-max activation function Hard limit activation function
+, -.
, . , -.
Hyperbolic activation function (Tansig)
5
Y= x
function (Pureline) Logistic sigmoid
Derivative
, . +, -.
Y=
Yj =∑1
. , 0
.1 234 ,
( !) " #
$1 , & ! ' 0* 0, & ! ) 0
5 5
5
Range
=1
(-inf,+inf)
= y(1-y)
(0,+1)
5 5
= 1-y2
5 5
(-1,1)
(0,+1)
=0
Not applicable
(0,+1)
2-9 Neural Network Training The learning of neural network may be called training the property that is of primary significance for a neural network, is the ability of the network to learn from environment, and to improve its performance through learning. Training is divided into three types: [2]
24
Chapter Two: Theoretical Part
2-9-1 Supervised Training: The learning process in which the teacher teaches the network by giving the network the knowledge of environment in the form of sets of the inputsoutputs pre-calculated examples, like training in athletics, training in a neural network requires a coach, someone that describes the neural network what should have produced as a response. From the difference between the desired response and actual response, the error is determined and a portion of it is propagated backward through the network. At each neuron in the network, the error is used to adjust the weights and threshold values of the neuron, so that the next time, the error in the network response will be less for the same inputs. Backpropagation (BP), general regression neural network (GRNN) and Genetic Algorithm (GA) are some of supervised training algorithms. [20]
2-9-2 Unsupervised Training: In unsupervised or self-organized learning, there is no external teacher or critic to oversee the learning process. Rather provision is made for a task independent measure of the quality of the representation; the free parameters of the network are optimized with respect to the measure. Once the network has become tuned to the statistical regularities of the input data, it develops the ability from internal representation for encoding features of the input and thereby to create the new class automatically. [36]
25
Chapter Two: Theoretical Part
2-9-3 Reinforcement Training/ Neurodynamic Programming Similar to supervised learning-instead of being provides with the correct output value for each given input; In reinforcement learning, the learning of an input and output mapping are performed through continued interaction with environment due to minimize a scalar index of performance. [20]
2-10 Weights: Each neurons has a specific weight and it directly affect on our input. Compared to a biological neurons quantity weight which is corresponding to strength of synaptic connection; weight values are associated with each vector and node in the network, and these values constrain how input data are related to output data. Weight values associated with individual nodes are also known as biases. These values are determined by the iterative flow of training data through the network, i.e., these are established during a training phase in which the network learns how to identify particular classes by their typical input data characteristics. [19] Relative to biological neurons, weight values are corresponding to the strength of synaptic connections; this is explained in figure (2-9), so the effect of (Xi) inputs on (Y) can be possibly determined by using its weight values. For example, cases like imprest-giving in the banks, the importance of salary and age of the person who takes the imprest can be determined by component weight then compared with output. [19] 6 !=∑:8; 78 9 8 ,
8; ∑:8; 78 9 8 or
=79 ……………. (2-12)
26
Chapter Two: Theoretical Part
7
Wi1
7
:
7:
wi1 :
< 78 9 8
wi2
Output Network
8;
win
Figure (2-9) represents the weight values corresponding to the strength of synaptic connections. [19]
2-11 Bias: Another parameter will be added to the net function and the bias improves the performance of the neural networks. This neuron lies in one layer, which is connected to all the neurons in the next layer, but none of the previous layers. Since the bias neuron emits 1 the weights, connected to the bias neuron, are added directly to the combined sum of the other weights. If the bias is present then the net is calculated as. [19] 6 != ∑:8; 78 9 8 +,
8; ∑:8; 78 9 8 + or =79 + …………… (2-13)
27
Chapter Two: Theoretical Part
7 7
wi1 :
< 7= 9&= $
wi2
Output Network.
8;
win 7:
Bias
Figure (2-10) the network with Bias [19]
2-12 Artificial Neurons Artificial neuron is a basic building block of every ANN. Its design and functionalities are derived from observation of a biological neuron that is basic building block of biological neural networks (systems) which includes the brain, spinal cord and peripheral ganglia. Similarities in design and functionalities can be seen in fig. (2-11). Where, the left side of a figure represents a biological neuron with its soma, dendrites, and axon, then the right side of a figure represents an artificial neural network with its inputs, weights, transfer functions, bias and output.[24]
28
Chapter Two: Theoretical Part
Figure (2-11 11) Biological and artificial neuron design. [24]
2-13 Difference between RNNs and FFNNs As described in the neural networks can be classified into two types; feed-forward forward neural networks (FFNNs) and Recu Recurrent rrent Neural Networks (RNNs). FFNNs differs from RNNs regarding to feedback connection between the neurons in the later. In FFNNs there are no any feedback connections between its neurons. In contrast RNNs allow feedback connections among its neurons at least one time, which is in that time the network topology can be very general; each neuron can be connected to each other, even to itself. It is allowing the presence of feedback connections between neurons, which has an advantage; it leads naturally to an analysis of the networks as dynamic systems. That means the state of a network at one moment in time depends on the state at previous moment in time. A recurrent neural network is a new part of artificial neural network (ANN), while connections between units are from a directed cycle with having loops in the networks. [3]
29
Chapter Two: Theoretical Part
Figure (2-12) shown the differencing between RNNs and FFNNs. [3]
2-14 Neural Networks with Algorithms. The perceptron can be trained by adjusting the weights of the inputs with Supervised Learning. In this learning technique, the patterns to be recognized are known in advance, and a training set of input values are already classified with the desired output output.. Before commencing, the weights are initialized with random values. Each training set is then presented for the perceptron in turn. For every input set the output from the perceptron is compared to the desired output. If the output is correct, no weights are altered. However, if the output is wrong, we have to distinguish which of the patterns we would like the result to be, and adjust the weights on the currently active inputs towards the desired result. [6]
2-15 Levenberg-Marquardt Marquardt algorithms (LMA) The Levenberg-Marquardt Marquardt algorithm, also known as the damped leastleast squares (DLS) method, is used to solve non non-linear linear least squares problems. These minimization problems arise especially in least squares curve fitting. While backpropagation is a steepest des descent cent algorithm, the LevenbergLevenberg
30
Chapter Two: Theoretical Part
Marquardt algorithm is a variation of Newton’s method. The advantage of Gauss–Newton over the standard Newton’s method is that it does not require calculation of second-order derivatives. The Levenberg-Marquardt algorithm trains an ANN faster (10–100 times) than the usual backpropagation algorithms. The Levenberg-Marquardt algorithm is used in many software applications for solving generic curve-fitting problems. However, as for many fitting algorithms, the LMA finds only a local minimum, which is not necessarily the global minimum. The LMA interpolates between the Gauss–Newton algorithm (GNA) and the method of gradient descent. The LMA is more robust than the GNA, which means that in many cases it finds a solution even if it starts very far off the final minimum. For well-behaved functions and reasonable starting parameters, the LMA tends to be a bit slower than the GNA. LMA can also be viewed as Gauss–Newton using a trust region approach. [43]
2-16 Derivation of Levenberg-Marquardt algorithm In this section, the derivation of the Levenberg-Marquardt algorithm will be presented in four parts: [8] 1- Steepest descent algorithm. 2- Newton’s method. 3- Gaussian-Newton’s algorithm. 4- Levenberg-Marquardt algorithm.
31
Chapter Two: Theoretical Part
Before the derivation, let us introduce some commonly used indices: • > Is the index of patterns, from 1 to>, where p is the number of patterns.
• ? Is the index of outputs, from 1 to @A , where @A is the number of outputs.
• i and j are the indices of weights, from 1 to 6, where 6 is the number of weights.
• Ψ Is the index of iterations. Other indices will be explained in related places. Sum square error (SSE) is defined to evaluate the training process. For all training patterns and network outputs, it is calculated by: E(x,w)=
F G
G E ∑H C34 ∑D 34 ,C,D
………………………… (2-14)
Where: x: is the input vector. w: is the weight vector. I,JG :
is the training error at output m when applying pattern > and it is
defined as:
I,JG :
= KI,JG - @I,JG ……………………………. (2-15)
32
Chapter Two: Theoretical Part
Where:
K: Desired output vector. @A : Actual output vector.
2-16-1 Steepest Descent algorithm The steepest descent algorithm is a first-order algorithm. It uses the firstorder derivative of total error function to find the minimum in error space. Normally, gradient g is defined as the first-order derivative of total error function (2-14): [8] ℊ=
MN( ,O) MO
=[
MN
MN
MO MO
….
MN
MOP
]T ……………. (2-16)
With the definition of gradient ℊin equation (2-16), the update rule of the steepest descent algorithm could be written as:
QR+ = QR – S ℊR …………………………………… (2-17)
Where:
S : learning constant rate between (0, 1). The training process of the steepest descent algorithm is asymptotic convergence. A round the solution, all the elements of gradient vector would be very small and there would be a very tiny weight change.
33
Chapter Two: Theoretical Part
2-16-2 Newton’s Method Newton’s method assumes that all the gradient componentsℊ , ℊ ,…..,
ℊP are function of weights and all weights are linearly independent: [8] ℊ " (Q1 , Q2 , … . , Q6 ) W 1 ] U ℊ2 " (Q1 , Q2 , … . , Q6 ) U ………………. (2-18) . V \ . U U ( ℊ " Q , T 6 P 1 Q2 , … . , Q6 )[
Where: , …. P are nonlinear relationships between weights and related gradient
components. Unfold each ℊ (i= 1,2,…,N) in equation (2-18) by Taylor
series and take the first-order approximation:
1 1 1 W ℊ1 ≈ ℊ1,0 $ MQ1 ∆Q1 $ MQ2 ∆Q2 $ ⋯ $ MQ6 ∆Q6 ] U U ℊ ≈ ℊ $ Mℊ2 ∆Q $ Mℊ2 ∆Q $ ⋯ $ Mℊ2 ∆Q U U 1 2 6 2 2,0 M Q1 M Q2 M Q6 …. (2-19) . V \ . U U M ℊ6 M ℊ6 M ℊ6 U U ℊ ≈ ℊ $ ∆ Q $ ∆ Q $ ⋯ $ ∆ Q 1 2 6 6,0 T 6 [
Mℊ
Mℊ
Mℊ
M Q1
M Q2
M Q6
By combining the definition of gradient vector g in equation (2- 16), it could be determined that: Mℊ2
MQ0
=
Ma
bc d bQ0
MQ0
=
ME N
MQ2 MQ0
………………………………… (2-20)
By inserting equation (2-20) to (2-19):
34
Chapter Two: Theoretical Part W ℊ1 ≈ ℊ1,0 $ MQ4E ∆Q1 $ MQ1MQ2 ∆Q2 $ ⋯ $ MQ1MQ6 ∆Q6 ] U U ℊ ≈ ℊ $ MEN ∆Q $ MEN ∆Q $ ⋯ $ MEN ∆Q U U 1 2 6 2 2,0 M Q1 M Q2 MQEE M Q2 M Q6 … (2-21) . V \ . U U U U ME N ME N ME N ℊ ≈ ℊ $ ∆ Q $ ∆ Q $ ⋯ $ ∆ Q 1 2 6 E 6 6,0 T [ M Q6 M Q1 M Q6 M Q2 MQe ME N
ME N
ME N
Comparing with the steepest descent method, the second-order derivatives of the total error function need to be calculated for each component of gradient vector. In order to get the minimum of total error function E, each element of the gradient vector should be zero. Therefore, left sides of the equation (2-21) are all zero, then: W 0 ≈ ℊ1,0 $ MQ4E ∆Q1 $ MQ1 MQ2 ∆Q2 $ ⋯ $ MQ1MQ6 ∆Q6 ] U U 0 ≈ ℊ $ MEN ∆Q $ ME N ∆Q $ ⋯ $ ME N ∆Q U U 1 2 6 2,0 M Q1 M Q2 MQEE M Q2 M Q6 …. (2-22) . V \ . U U U U ME N ME N ME N T0 ≈ ℊ6,0 $ MQ6 MQ1 ∆Q1 $ MQ6MQ2 ∆Q2 $ ⋯ $ MQeE ∆Q6 [ ME N
ME N
ME N
By combining equation (2-16) and (2-22) W − MQ4 " −ℊ1,0 ≈ MQ4E ∆Q1 $ MQ1MQ2 ∆Q2 $ ⋯ $ MQ1MQ6 ∆Q6 ] U U − MN " −ℊ ≈ ME N ∆Q $ MEN ∆Q $ ⋯ $ MEN ∆Q U U 1 2 6 2,0 MQE M Q1 M Q2 MQEE M Q2 M Q6 (2-23) . V \ . U U U MN U ME N ME N ME N − " − ℊ ≈ ∆ Q $ ∆ Q $ ⋯ $ ∆ Q 1 2 6 E 6,0 T MQe [ M Q6 M Q1 M Q6 M Q2 MQe MN
ME N
ME N
ME N
There are (N) equations for (N) parameters so that all (∆Q )can be
calculated. With the solutions, the weight space can be updated iteratively. Equations (2-23) can be also written in matrix form
35
Chapter Two: Theoretical Part M N M N M N k − MQ4 n k MQ4E MQ1MQ2 … MQ1MQ6 n k ∆Q n j MN m j MEN ME N m j ∆Q m ME N − j m j g … h= MQE = MQ MQ MQ E … MQ MQ m*j . m… (2-24) 6m 2 j … m j 1 2 E . m ME N ME N m j ℊ6 j MN m j MEN … i∆QP l MQ E l i− l iM Q M Q M Q M Q
ℊ1 ℊ2
MN
E
MQe
6
E
1
E
6
2
e
Where the square matrix is Hessian matrix: k MQ4E MQ1 MQ2 … j ME N ME N ℋ=j MQ MQ MQ E … j 1 2 E ME N j ME N iM Q6 M Q1 M Q6 M Q2 ME N
ME N
ME N
n m ME N m…………………….. (2-25) M Q2 M Q6 m ME N m … El MQe
M Q1 M Q6
By combining equation (2-16) and (2-25)
-ℊ= ℋ∆Q ……………… (2-26)
So
∆Q = -ℋ ℊ ………….. (2-27)
Therefore, the update rule for Newton’s method is: QR+ = QR - ℋR ℊR …… (2-28) As the second-order derivatives of total error function, Hessian matrix
ℋgives the proper evaluation on the change of gradient vector. By
comparing equation (2-17) and (2-28), one may notice that well matched step size are given by the inverted Hessian matrix.
36
Chapter Two: Theoretical Part
2-16-3 Gaussian-Newton Algorithm If Newton’s method is applied for weight updating, in order to get
Hessian matrix ℋ, the second-order derivatives of total error function have
to be calculated and it could be very complicated. In order to simplify the calculating process, Jacobian matrix pis introduced as: [8]
4,4 4,4 4,4 k MQ4 MQE … MQe n j Mℯ4,E Mℯ4,E Mℯ4,E m … j MQ4 MQE MQe m … … … j Mℯ Mℯ m Mℯ4,s 4,s 4,s j MQ MQ … MQ m 4 E e m……………….. (2-29) … … … |p|=j Mℯ Mℯ Mℯ H,4 H,4 H,4 j m … MQe m j MQ4 MQE j MℯH,4 MℯH,4 … MℯH,4 m MQe m 4 j MQ4 MQ … … … jMℯH,s MℯH,s MℯH,s m … i l
Mℯ
Mℯ
Mℯ
MQ4 MQE
MQe
By integrating equation (2-14) and (2-16), the elements of gradient vector can be calculated as: ℊi =
MN
MQ2
=
4 E
F
G E Mt ∑H C34 ∑DG 34 ,C,DG u
MO2
= ∑xI; ∑vGG; t w
M,C,DG MQ2
I,vG u…..
(2-30)
matrix |p| and gradient vector ℊ would be
Combining equation (2-29) and (2-30), the relationship between Jacobian ℊ = |p|ℯ ………………………… (2-31)
Where error vector e has the form
37
Chapter Two: Theoretical Part
ℯ, k ℯ, j … jℯ j ,y ℯ=j … j ℯx, j ℯx, j … iℯx,y
n m m m m …………………….. (2-32) m m m l
Inserting equation (2-14) into (2-24), the element at & th row and =th column of Hessian matrix can be calculated as: z ,8 =
ME N
MQ2 MQ0
=
1 } 2 {2 t2 ∑~>"1 ∑@ ?} "1 >,| u
MQ2 MQ0
Where ,8 is equal to
,8 = ∑xI; ∑vGG; w
= ∑xI; ∑vGG; w
M,C,DG MQ2
*
M,C,DG MQ0
ME ,C,DG
…………….. MQ2 MQ0 I,vG
+ ,8 …. (2-33)
(2-34)
As the basic assumption of Newton’s method is that ,8 is closed to zero, the
relationship between Hessian matrix ℋ and Jacobian matrix |p| can be
rewritten as:
ℋ = |p p|………………… (2-35)
By combining equation (2-28), (2-31) and (2-35), the update rule of the Gauss-Newton algorithm is presented as: QR+ = QR – (p p )-1 p ℯk ………………..(2-36) The advantage of the Gauss-Newton algorithm over the standard Newton’s method in equation (2-31) is that the former does not require the calculation 38
Chapter Two: Theoretical Part
of second-order derivatives of the total error function, by introducing Jacobian matrix instead. However, the Gauss-Newton algorithm still faces the same convergent problem can like the Newton algorithm for complex error space optimization. Mathematically, the problem can be interpreted as the matrix (|p p|) may not be invertible.
2-16-4 Levenberg-Marquardt Algorithm Rule In order to make sure that the approximated Hessian matrix (p p ) is
invertible,
Levenberg-Marquardt
algorithm
introduces
another
approximation to Hessian matrix: [8]
ℋ = p′p +SI …………………………. (2-37)
Where:
S : is always positive, called combination coefficient. I: is identity matrix. From equation (2-37), one may notice that the elements on the main diagonal of the approximated Hessian matrix will be larger than zero. Therefore, with this approximation equation (2-37), it can be sure that matrix ℋ is always invertible.
By combining equation (2-36) and (2-37), the update rule of LevenbergMarquardt algorithm can be presented as: QR+ = QR – (p p + SI)-1p k ℯk ……………………. (2-38) As the combination of the steepest descent algorithm and the Gauss-Newton
39
Chapter Two: Theoretical Part
algorithm, the Levenberg-Marquardt algorithm switches between the two algorithms during the training process. When the combination coefficient S
is very small (nearly zero), equation (2-38) is approaching to equation (234) and Gauss-Newton algorithm is used. When combination coefficient S
is very large, equation (2-38) approximates to equation (2-21) and the steepest descent method is used. If the combination coefficient S in equation
(2-38) is very large, it can be interpreted as the learning coefficient in the steepest descent method (2-17):
= ………………… (2-39)
Table (2-2) summarize the update rule for various algorithm (Specifications of Different Algorithm): [8] Algorithms
Update rule
Convergence
EBP algorithm
Q k+1= Q k – ℊk
Stable, slow
Newton algorithm Gauss-Newton algorithm
Q k+1= Q k – - ℋR ℊR
QR+ = QR – (p p )-1 p ℯk
Levenberg-Marquardt algorithm*** QR+ = QR – (p p + SI) p k ℯ k -1
Unstable, fast Unstable, fast Stable, fast
40
Chapter Two: Theoretical Part
2-17 Algorithm Enforcement In order to implement the Levenberg-Marquardt algorithm for Neural Network training, two problems have to be solved: how does one calculate the Jacobian matrix, and how does one organize the training process iteratively for weight updating. In this part, the enforcement of training with the Levenberg-Marquardt algorithm will be introduced in two parts. [8] 1- Calculation of Jacobian matrix. 2- Training process design.
2-17-1 Calculation of Jacobian Matrix In the computation followed, j and k are used as the indices of neurons, from 1 to number of neurons contained in a topology; I is the index of neuron inputs, from 1 to number of inputs and it may vary for different neurons. As an introduction of basic concepts of neural network training, let us consider a neuron j with number of inputs, as shown in figure (2-13). If neuron j is the first layer, all its inputs would be connected to the inputs of the network, otherwise, its inputs can be connected to outputs of other neurons or to networks input if connections across layers allowed. Node y is an important and flexible concept. It can be yj,i, meaning the ith input of neuron j. It also can be used as yj to define the output of neuron j. In the following derivation, if node y has one index then it is used as a neuron output node, but if it has two indices (neuron and input), it is a neuron input node. [8]
41
Chapter Two: Theoretical Part
The output node of neuron j is calculated using
8 = fj(Netj) …………………… (2-40)
fj : the activation function of neuron j. Netj : is the sum weighted input nodes of neuron j. Netj=∑: 8; 98, 8, +bj………. (2-41) Where:
8, : is the ith input of neuron j, weighted by 98, . bj: is the bias weight of neuron j.
8,
wj,28,
8,
wj,ni wj,o 8,: 8,:
wj,1 wj,2 wj,i wj,ni-1
8 ( !8 )
yj
,8 (8 )
Om
wj,ni wj,0
+1
Figure (2-13) Represented connection between neurons of the network. “
Prepared by researcher”
42
Chapter Two: Theoretical Part
Where:
8, : represent network inputs. Fm,j(yj) : is the nonlinear relationship between the neuron output node yj Om: is the network output. Using equation (2-41), one may notice that derivative of (Netj) is: M:, 0 MO0,2
" 8, ……………………….. (2-42)
And slope of activation function fj is =
M0
M:, 0
=
M8(:, 0 ) M:, 0
……. ………….. (2-43)
Om = Fm,j(yj) …………………………. (2-44) The complexity of this nonlinear function Fm,j(yj) depends on how many other neurons are between neuron j and network output m. if neuron j is at network output m.
Then Om=yj and
,8 (yj) =1, where
,8 is the derivative of nonlinear
relationship between neuron j and output m.
The elements of Jacobian matrix in equation (2-29) can be calculated as: M,C, MO0,2
=
M (5C,- wC, ) MO0,2
=-
MwC, MO0,2
=-
MwC, M0
M0
M:, 0
*
*
M:, 0 MO0,2
……… (2-45)
43
Chapter Two: Theoretical Part
Combining with equations (2-40) through (2-42) can be rewritten as: M,C, MO0,2
= -
8 8, ………………….. (2-46)
Where:
8 : is the derivative of nonlinear function between neuron j and
output m.
The computation process for Jacobian matrix can be organized according to the traditional backpropagation computation in first-order algorithms (like the error backpropagation EBP algorithm). But there are also differences between them. First of all, for every pattern, in the EBP algorithm, only one backpropagation process is needed, while in the Levenberg-Marquardt algorithm the backpropagation process has to be repeated for every output separately in order to obtain consecutive rows of the Jacobian matrix. Another difference is that the concept of backpropagation of parameter
has to be modified. In the EBP algorithm, output errors are parts of the parameter:
8 = 8 ∑y ;
8
………….. (2-47)
In the Levenberg-Marquardt algorithm, the parameters are calculated for each neuron j and each output m.
,8 = 8
8 …………………… (2-48)
By combining equations (2-46) and (2-48), elements of the Jacobian matrix can be calculated as: M,C, MO0,2
= - ,8 8, ……………. (2-49) 44
Chapter Two: Theoretical Part
There are two unknowns is equation (2-49) for the Jacobian matrix computation. The input,8, , can be calculated in the forward computation
(signal propagating from inputs to outputs); while ,8 is obtained in the
backward computation, which is organized as errors backpropagating from output neurons (Output layer) to network inputs (Input layer). At output neuron m(j)=m , ,8 = .
For a given pattern, the forward computation can be organized in the following steps: 1- Calculate network value, Slopes, and Output for all neurons in the first layer: 6 !8 = ∑: ; 98, $ 98,v …….. (2-50)
8 = 8 (6 !8 ) …..……………… (2-51)
8 =
M04
MP, 04
…..…………………… (2-52)
Where:
: inputs the network.
j: the index of neurons in the first layer. 2- Use the outputs of the first layer neurons as the inputs of all neurons in the second layer, do a similar calculation for Network values, slopes, and Outputs:
45
Chapter Two: Theoretical Part 6 !8 = ∑: ; 98, $ 98,v ………….. (2-53)
8 = 8 (6 !8 ) …..………………….. (2-54)
8 =
M0E
MP, 0E
…..……………………….. (2-55)
3- Use the outputs of the second layer neurons as the inputs of all neurons in the output layer (third layer), do a similar calculation for Network values, slopes, and Outputs:
6 !8 = ∑: ; 98, $ 98,v …………… (2-56)
8 = 8 (6 !8 ) …..…………………… (2-57)
M0 8 = MP, 0
…..………………………... (2-58)
After the forward calculation, node array y and slope array can be
obtained for all neurons with the given pattern.
With the results from the forward computation, for a given output j, the backward computation can be organized:
4- Calculate error at the output j and the initial as the slope of output j: 8
= 8 − ?8 ……………………. (2-59)
8,8 = 8 ……………………… (2-60) 8,R = 0 ……………………….. (2-61)
46
Chapter Two: Theoretical Part
Where:
8 : Desired output. ?8 : Actual output.
8,8 : Self-backpropagation.
8,R : Backpropagation from other neurons in the same layer (output
layer).
5- Back propagate from the inputs of the third layer to the outputs of
the second layer.
8,R = 98,R 8,R ……………. (2-62)
Where: k: the index of neurons in the second layer. 6- Back propagate from the outputs of the second layer to the inputs of the second layer. 8,R = 8,R R …………………….. (2-63)
7- Back propagate from the inputs of the second layer to the outputs of the first layer.
8,R = ∑: ; 98, 8, ……………... (2-64)
8- Backpropagate from the outputs of the first layer to the inputs of the first layer.
47
Chapter Two: Theoretical Part 8,R = 8,R R ………………….. (2-65)
For the backpropagation process of other outputs, the steps (4)-(8) are repeated.
2-17-2 Training Process Design With the update rule of the Levenberg-Marquardt algorithm in equation (2-38) and the computation of Jacobian matrix, the next step is to organize the training process. According to the update rule, if the error goes down, which means it is smaller than the last error, it implies that the quadratic approximation on total error function is working and the combination
coefficient S could be changed smaller to reduce the influence of gradient
descent part (ready to speed up). On the other hand, if the error goes up, which means it’s larger than the last error, it shows that it’s necessary to follow the gradient more to look for a proper curvature for quadratic approximation and the combination coefficient S is increased. [8]
48
Chapter Two: Theoretical Part
Figure (2-14) Represent training using Levenberg-Marquardt algorithm. [8] Where:
9R : The current weight.
9R+ : The Next weight.
R+ : The current total error. R : The last total error.
49
Chapter Two: Theoretical Part
2-18 some types of measure important for choose the best network: 2-18-1 Akaike Information Criterion (AIC) The statistical measure named by (Akaike Information Criterion), which is one frequently used criterion for nonlinear model identification. AIC formula is given by: [52] AIC= -2logL+2m ……………………………………….. (2-66) Where: m: is the number of weights (parameters) used in RNN, and also: -2logL = -2 [∑: ; [log(2) $ log , +
( )^E E
] ………. (2-67)
Where:
, : The error of variance.
: Desired output to the network.
: The network output at time (t). n: The number of input observation to train the network. Or AIC = n*ln (SSE / n) + 2m …………………………………. (2-68)
50
Chapter Two: Theoretical Part
Such that: n: The number of training cases. m: Denotes number of parameters of weights in suggested (RNN). m = n (nh +1) +2nh +1 nh: The Number of nodes in hidden layer(s). The measure of Fitness model is given by:
Fitness = 1 / Testing set (MSE), 0≤
&! ) ∞…….(2-69)
The decision of disability for these (RNNs) was made with respect to the accuracy measure values (Fitness and AIC) for each design, maximum fitness corresponding minimum AIC value indicates the best RNN architecture. These measure of goodness of fit were used for all (RNN), which were candidate during this study.
2-18-2 Mean Square Error (MSE) The mean squared error (MSE) or mean squared deviation (MSD) of an estimator measures the average of the squares of the errors or deviations, that is, the difference between the estimator and what is estimated. MSE is a risk function, corresponding to the expected value of the squared error loss or quadratic loss. The difference occurs because of randomness or because the estimator doesn't account for information that could produce a more accurate estimate. [18]
51
Chapter Two: Theoretical Part
MSE = ∑: ;( − ) …………………… (2-70)
:
2-18-3 Coefficient of determination (R2) Is a number that indicates how well data fit a statistical model sometimes simply a line or a curve. An R2 of 1 indicates that the regression line perfectly fits the data, while as R2 of 0 indicates that the regression line does not fit the data at all. This latter can be because the data is more nonlinear than the curve allows, or because it is random. A data set has (n) values marked x1,x,,….,xn(collectively known as yi or as a vector x = [x1,x,,….,xn]n), each associated with a predicted value ? , ? , … , ?: (known as ? , or sometimes 7 , as a vector o). [16] The residuals such as:
" 7 − ? …………………….. (2-71)
The mean of the observed data: 7 =
∑4234 2 :
……………………….. (2-72)
Then the variability of the data set can be measured using three sums of squares formulas: 1- The total sum square.
SStotal=∑: ;(7 − 7 )2………………. (2-73)
52
Chapter Two: Theoretical Part
2- The regression sum square.
SSregression= ∑: ;(? − 7 )2………………. (2-74)
3- The residual sum square.
SSresidual=∑: ;(7 − ? )2 = ∑: ;
……. (2-75)
The most general definition of the coefficient determination (R2). R2 = 1-
2G ¡ D ¡
……………………… (2-76)
Or R2 =
¢2D£ D ¡
…………………….. (2-77)
2-19 Time series analysis and prediction The analysis of experimental data that have been observed at different points in time leads to new and unique problems in statistical modeling and inference. The obvious correlation introduced by the sampling of adjacent points in time can severely restrict the applicability of the many conventional statistical methods traditionally dependent on the assumption that these adjacent observations are independent and identically distributed. The systematic approach by which one goes about answering the mathematical and statistical questions posed by these time correlations is commonly referred to as time series analysis.
[23]
In this thesis time-series
will be used to build models that can be used for prediction. The time-series have a number of properties.
53
Chapter Two: Theoretical Part
2-19-1 Time-series Time series is one of the most common types of series variables. (Pyle, 1999) mention that they usually have at least two dimensions where one dimension represents some kind of continuous time and the other dimensions often represent some variables that very over time. In non-series multivariable measurements, the order is very important, unless a dataset is ordered it is not a series. Pyle points out that in a series, one of the variables is monotonic and is called the displacement variable. This variable is always either increasing or decreasing and represent time.[22]
2-19-1-1 Time series Analysis There are three goals with time series analysis: [15]
1- Modeling: The aim of modeling is a capture the long-term behavior of a system and makes an accurate description of these.
2- Prediction: The goal of prediction or forecasting is to do an accurate prediction of the short-term behavior of the system. Gershenfeld (1994) et al, argue that the short-term and long-term behaviors not necessarily are identical. [12]
3- Characterization: The goal of system characterization, tries according to Gershenfeld (1994) et al, to capture systems fundamental properties with little or no prior knowledge of the system. For example is the amount of randomness or the number of degrees freedom. [12]
54
Chapter Two: Theoretical Part
2-19-1-2 Problems with time series Analysis Pyle (1999) argues that series data have many of the problems non-series data have. Series data also have a number of special problems such as:
1- Outlier’s data Outliers are variables that have a value that is far away from the rest of the values for that variable. [38]
2- Noisy data Noise is a simply a distortion to the signal and is something integral to the nature of the world, not the result of a bad recording of values. Noise is that do not follow any pattern that is easily detected. [22]
3- Missing values or null values Missing values can cause big problems in series data and series modeling techniques. They are more sensitive to miss values than non-series modeling techniques there many different methods for “repairing” series data with missing values such that multiple regression and autocorrelation. [22]
2-19-1-3 Time series prediction The desire to predict the future and understand the past. This drives the search for rules that describes observed phenomena. If the underlying equations are known, they could in principle be solved and used to forecast the outcome of a given input situation. Another more difficult problem
55
Chapter Two: Theoretical Part
when the underlying equations not are known. Then, not only have the rules to be known, but also the actual state of the system. The roles can be found by looking at regularities in the past. Use the terms understanding and learning to describe two approaches for analyzing time series. With understanding mean that the analyzing is based on explicit mathematical insight into the system behavior, and with learning the analysis method is based on algorithm that can emulate the behavior of time series. [13]
2-19-2 Methods for Time series prediction There are several different methods for time series prediction:
2-19-2-1 linear models Linear models in time series have two particularly desirable features: they are relatively easy to understand and they can be implemented in a straightforward manner. The problems with these models are that they may be inappropriate for many systems, especially when the complexity grows.[12]
2-19-2-2 Moving Average Models (MA) Suppose that they have a linear and causality system. They also are given a univariate external series {¤} as input. They want to modify the input series to produce another series {7}. By the causality, the present values of x is dependent on the present value and the (¥) past values of (¤). The relationship between {¤ } and {7 t} is: [4]
7 t = ∑P :;J n¤ : = o ¤ + 1¤ + …. + N¤ P …………… (2-78) 56
Chapter Two: Theoretical Part
The output is generate by coefficients b0,….,bn from the external series. This is called Nth-order moving average model, MA (¥). The model is also called finite impulse response (FIR) because an input impulse at time t only effect the output values for t…. t+q, this means that the output values always become zero N time steps after the input values go to zero.
2-19-2-3 Auto regressive Models (AR) Sometimes the modeled system is not only dependent on the input but also on the internal states or outputs. Moving average or finite impulse response has no feedback from the internal states or the output and thus (MA) models can only transform an input that is presented from an external source. We say that the series is externally driven. If they do not want this external drive we need to provide some feedback or memory to model the internal dynamics of the series. [4]
7 t = ∑y ; ∅ 7 +¤ = ∅ 7 t-1 +…. +∅y 7 y +¤ …………… (2-79)
This is an Mth-order autoregressive model AR (p) or an infinite impulse response (IIR) filter because if the input goes the zero the output can still continue. The value of {¤ } can either be a controlled input or some kind of
noise. There is a relationship between MA and AR models, namely “any AR model can be expressed as an MA model of infinite order”. [4]
57
Chapter Two: Theoretical Part
2-19-2-4 Mixed Autoregressive and Moving Average Models (ARMA) If we combine both the AR (>) and MA (¥) models we get the ARMA
(>, ¥) model:
7 t = ∑; ∅ 7 +∑:;J n¤ : …………………… (2-80) I
§
With the ARMA model we can model most linear systems whose output depends on both the inputs and on the outputs. ARMA models are used to model various kinds of linear system. [4]
2-19-2-5 Non-linear Models Although linear models is suitable for many time series they perform worse on time series generated from non-linear data sources.
[31]
A neural
network can be viewed as a transformation function that maps or relates data from an input data set to an output data set. A neural network consists of a number of weights that is used to determine the output for a certain input. The network can be trained to do the mapping by presenting a number of inputs with their corresponding outputs and let a learning function adjust the weights. [10]
58
Chapter Two: Theoretical Part
2-20
Non-linear
autoregressive
moving
average
model
(NARMA): In this case focus on nonlinear of ARMA model for recurrent neural network and how to apply (NARMA) model in RNN. Let’s have a simple non-linear generalization of ARMA (p,q) model: [42] 7 " ¨7 , 7 , … , 7 I , © , © , … , © § ª $ © …… (2-81) Where:
7 : denoted the set of observation depend on time (t). © : denotes random noise, independent of past (7 ).
: is an unknown smooth function with the assumption the best (MSE). The prediction of equation above is:
7 " « (7 , 7 , … , 7 I , ©̂ , ©̂ , … , ©̂ § ) ………….. (2-82)
9 8 : denotes the coefficients of a full matrix weights. : denotes the activation function.
If the model « (7 , 7 , … , 7 I , ©̂ , ©̂ , … , ©̂ § )is chosen, then the RNN approximate it as.
I § 7 = ¨7 , 7 , … , 7 I ª= ∑ ; (∑8; 9 8 7 8 $ ∑8; 9 8 (7 8 −
7 8 )) …………………………………………………. (2-83)
59
Chapter Two: Theoretical Part
This model is a special case of the fully interconnected RNN 7 = ∑ ; (∑8; 9 8 7 8 ) …………………………. (2-84) I
2-21 Recurrent Neural Networks versus Feedforward Models: When using neural Networks in a dynamical system context it is important to decide about the model structure. In the context of Input/ Output models it’s important to make a distinction between NARX (Nonlinear Autoregressive with exogenous) and NOE (Non-linear Output Error) models. In NARX models one has. [14]
@ = (@ , @ , … , @ § , ® , ® , … , ® § )……… (2-85)
@ : denotes the true output at discrete time instant (!). ® : denotes the input at time (!).
@ : denotes the estimated output at time (!).
¥: denotes the number corresponds to the order of the system. In NOE models one has.
@ = (@ , @ ,…, @ § , ® , ® , … , ® § )…….. (2-86)
Note that one has a recursion now on the variable (@ ) in constraint with the
NARX model. From a neural networks perspective, the NARX model may be considered as a FFNN model, while the NOE model is Recurrent Neural Network.
60
Chapter Two: Theoretical Part
Then, models for time series prediction are closely related to these models by omitting the input variable (®), one obtains then:
@ + = (@ , @ , … , @ § )……………………… (2-87)
Which is parameterized by an MLP as
@ + = tanh ( ³[@ , @ , … , @ § ] $ µ )……….. (2-88)
³: Nonlinear function.
It is not necessary that the past values(@ , @ , … , @ § ) are subsequent in time certain values could be omitted or values as different time scales could
be taken. In order to generate prediction, the true values (@ )are replaced then by the estimated values (@ ) and the iterative is generated by the RNN. @ + = tanh ( ³[@ , @ , … , @ § ] $ µ )…………………. (2-89)
For a given initial condition. Instead of Input/ Output models one may also take discrete time non-linear state space descriptions.
+ " ( , ® )* ¶ ………………………………… (2-90) @ " ¨ ª
Recurrent Neural Network models are e.g. used in control application, where one first identifies a model and then design a controller based upon the identified model and applies it to the real system, either in a nonadaptive or adaptive setting. When using neural networks in a dynamical systems context, one should be aware that even very simple Recurrent Neural Networks can lead to complex behavior such as chaos. In this sense
61
Chapter Two: Theoretical Part
stability issues of multi-layer RNNs are important e.g. towards applications in signal processing and control. The training also more complicated than for FFNNs. In the RNN case a cost function is defined on a dynamical system (iterative system) which leads to more complicated analytic expressions for the gradient of the cost function. [14]
2-22 Forecasting versus Prediction: Forecasting is the process of making predictions of the future based on past and present data and analysis of trends. A commonplace example might be estimation of some variable of interest at some specified future data. Prediction is a similar, but more general term. Both might refer to formal statistical methods employing time series cross section data or alternatively to less formal judgmental methods. Prediction can only be made when the accuracy of the prediction process can be characterized in terms of historic data used to compare a priori predicted outcomes to the actual outcomes. A priori is italicized because once one has seen the outcomes; any changes to the prediction process will generally require re-characterization of the error using data that has not been seen. Then the difference between prediction and forecasting is independent of the prediction process. One may use human instincts to make predictions. As long as the error associated with the instinctive prediction process can be characterized on a consistent basis statistically, confidence levels on the error can be produced. On the other hand, one can use large quantities of historic data to optimize coefficients in an intricate mathematical model that generates future outcomes without distinguish the error. [26]
62
Chapter Three: Application Part
3-1 Introduction In this chapter the researcher tried to suggest the best architecture and best design for the recurrent neural network (RNN) to view the contains of the suggested RNN about the number of layers of hidden layer and number of Nodes in each hidden layer, also to determine the best activation functions between layers that makes the Recurrent Neural Network have the best performance for the data under consideration that makes the model recognize all the complex patterns for the non-linear time series-cross section data.
3-2 Recurrent Neural Network Design 3-2-1 Data Description: The data were collected from the province of Sulaimani / Directorate of control and communication for electricity during the January of 2013 to July of 2015 in the average of daily power energy (load and demand) as (940 consecutive Obs.) as at time series (t=1,2,….,940). The data is measured by Ampere (A). The sample of this data is showed in Table (3-1) and in Table (B) in appendices data can be seen completely in detail.
63
Chapter Three: Application Part
Table (3-1) the sample of data used to application Load
Demand
Load
Demand
Load
Demand
Load
Demand
Feeder
Feeder
Feeder
Feeder
Feeder
Feeder
Feeder
Feeder
1
136.54
136.31
76
89.38
89.27
151
82.34
82.62
226
117.55
117.32
2
139.16
137.83
77
104.88
104.94
152
91.12
91.13
227
110.67
110.64
3
137.75
136.29
78
109.11
109.42
153
94.86
94.85
228
116.95
116.91
4
132.93
131.95
79
107.08
107.04
154
96.13
96.11
229
118.64
118.03
5
136.17
135.84
80
89.17
89.14
155
99.39
99.40
230
114.26
112.28
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
301
80.97
80.49
376
149.88
145.56
451
81.71
81.63
526
104.18
103.75
302
79.98
79.83
377
150.66
149.49
452
87.38
87.32
527
95.84
95.36
303
79.57
79.53
378
151.01
146.94
453
95.66
100.65
528
84.94
85.20
304
78.47
78.38
379
149.03
146.44
454
115.13
115.62
529
89.14
89.50
305
82.70
82.60
380
143.99
140.00
455
114.69
113.38
530
94.34
94.58
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
676
111.65
108.21
751
151.01
147.43
826
82.20
81.94
936
115.44
114.05
677
110.13
107.15
752
146.63
143.48
827
79.11
78.75
937
113.78
112.74
678
110.73
108.61
753
148.01
145.36
828
74.46
74.43
938
117.45
116.90
679
108.27
106.25
754
149.25
145.56
829
69.83
69.76
939
119.00
118.96
680
107.34
106.30
755
147.18
143.01
830
76.46
76.55
940
116.07
113.46
No.
No.
No.
No.
64
Chapter Three: Application Part
3-2-2 The Application Steps of Recurrent neural Networks: This part includes the application for creating Recurrent Neural Networks (RNN) for time series prediction. The Matlab software (R2014a V.8.3.0.532) has been used to apply (RNN) for data that described above. The application of Recurrent Neural Network for time series prediction in this thesis was done with the following steps:
First step: In this case identify data within MATLAB software, because we need two types of data in (RNN) (input data is demand on power energy(Xt; t= 1,2,3,4,…..,940)), and target data is load power energy (Yt; t= 1,2,3,4,…..,940)), the data that we have contains two types (Demand power and Load power) in this thesis the input data as the Demand power (Xt) and target data as a Load power (Yt).
Second step: Normalization data The goal of this step is to normalize the data and make them bounded between (-1, 1), or (0, 1) this coding depend on the behavior of the type of ANN or RNN that we make use of, especially if the data under consideration contains complex patterns as we have in our data under consideration it may be more flexible to use the first type of normalization above. This process is only for coding the observations of time series data to make them understandable inputs for all layers to the recurrent neural network layers. The equation below is a type of normalization for input time series data in the range (-1, 1). Z= (
௫ି୫୧୬ (௫) ୫ୟ୶(௫)ି୫୧୬ (௫)
∗ 2) − 1…………….. (3-1).
Where 65
Chapter Three: Application Part
Z: normalize data. x= origin data. Max(x): Maximum value of data,
Min(x): Minimum value of data.
Third step: Data Partitioning In artificial neural networks generally two types of partitioning can be used the first type is named by sequential partition that divided the data sequentially from first observation to the last with the order themselves using the proportion that the researcher suggested it as (%70) for training set, and (%15) for each testing and validation set respectively the second type is randomly partitioning as the researcher used it in this thesis. The random partition is better than the other because the only random partition can recognize the complex patterns in prediction or forecasting models. But the only disadvantage of the random partition is the researcher can’t return to the steps of application during the process that is because of the random choose of sets which are made by the software in the partition. Then numerically the partitions for the ratios are as follow. The first set for training the network with input data equal to (658) observations, the second set for testing the network with input data equal (141) observations and the third set for validation chosen with input data equal (141) observations.
Forth step: Create the Network architecture In this case the structure of the recurrent neural network that contains three layers (Input layer, Hidden layer(s) and output layer). At this stage we choose the best way for the performance of the network, two important measures the first is the akaike information criterion (AIC), and the second is the fitness coefficient value for choosing best recurrent neural network having the best performance that recognized all complex patterns exists, which depends on
66
Chapter Three: Application Part
minimum (AIC) and maximum (fitness), in this study the best RNN chosen to analyze data, two hidden layers network is used, which is explained in table (32) and figure (3-1) below. Table (3-2) Represent the best architecture of (RNN).
Layers
Nodes
Activation function
Input layer
1
---------------
5
Tansig (hyperbolic)
10
Tansig (hyperbolic)
1
Purelin (linear)
Hidden layers Output layer
Figure (3- 1) represent the best architecture RNN model (1-5-10-1). The best Network is [1-5-10-1], depend on Maximum fitness, Minimum AIC and mean square error for (training, testing, validation and overall data set). By using applying the activation function between layers and change the number of nodes between layers we get the best model is [Tansig1Tansig2Purelineoutput]. MSEtr
MSEts MSEval MSEall
0.000478 0.000239
0.00078
0.0015
R2tr
R2ts
R2val
R2all
0.99905
0.99902
0.99946
0.9991
Where: MSEtr, MSEts, MSEval,MSEall: Mean Square Errors for (Training, Testing, Validation, and all data set).
67
Chapter Three: Application Part
R2tr, R2ts, R2val, R2all: Coefficient of determinations for (Training, Testing, Validation, and all data set).
Fifth step: Training Network In this case during training suggested the Recurrent ecurrent Neural Network, the data would be analyzed and change weights among nodes to reflect dependencies and patterns. In this section we made use of training algorithm. Then we choose cho the best algorithm named by (Levenberg (Levenberg-Marquardt) which is explained in figure (3-2) 2) that shows the best training state state. It is clear that the best efficiency is occurred in epoch 12.. The learning function of learning data is shown in repetition 12 in Fig (3 (3-2)
Figure (3 (3- 2) show the training state. The variation of the gradient error (0.00066116) and validation checks at epoch (12) equal to (6).The diagram of learning errors, assessment errors and test errors and the best training performance with the best validation performance for (RNN), show the figure (3 (3-3).
68
Chapter Three: Application Part
In this figure below the best of training network performance is (0.00028243) at epoch (6) because the minimum global located at epoch (6).
Figure (3 (3-3) represent the training performance.
The performance for this model is below. MSE= 0.0015 for all model. From table (3-3)) represent the sample of finding the best architecture of RNN models and in Table (A) in appendices can be seen completely in detail. Clearly shown the number of trail to find the best per performance formance of model that depends on some importance scale cale comparable models are (MSE, AIC, R2, and Fitness Model). And from rom the table (3 (3-4) that represents finding the best of the best stages of architecture model in (RNN) for data under consideration. By using all techniques such as changes of (number of nodes, hidden layers and activation function).
69
Chapter Three: Application Part
From table (3-3) represents finding the best architecture of RNN model
R2ts
R2val R2all MSEtr
MSEts MSEval
0.998
0.999
0.999
0.998
0.00081
0.00035
0.998
0.999
0.998
0.998
0.00077
0.998
0.999
0.999
0.998
0.999
0.999
0.999
0.999
0.995
0.998
AIC
0.00042
0.0016
-4156.04
2844.707
17
0.00054
0.00037
0.0017
-4116.15
1847.37
10
0.00075
0.00049
0.00049
0.0017
-4116.15
2038.944
40
0.999
0.00045
0.002
0.00042
0.0028
-3717.81
500
18
0.998
0.998
0.00078
0.00043
0.00043
0.0016
-4086.04
2299.221
14
0.999
0.998
0.998
0.00074
0.00061
0.00047
0.0018
-4008.54
1649.811
25
0.999
0.999
0.999
0.999
0.00048
0.00024
0.00078
0.0015
-4234.51
4176.063
10
0.999
0.994
0.999
0.998
0.00075
0.00043
0.00059
0.0018
-4018.54
2334.485
11
0.999
0.998
0.993
0.998
0.00077
0.00039
0.00047
0.0016
-4096.04
2538.2
166
0.998
0.999
0.999
0.998
0.0008
0.00042
0.00038
0.0016
-3976.04
2389.772
11
0.998
0.999
0.998
0.998
0.00077
0.00038
0.00055
0.0017
-3936.15
2647.113
13
0.999
0.994
0.999
0.998
0.00075
0.00051
0.00049
0.0018
-3898.54
1970.172
101
0.998
0.999
0.999
0.998
0.00076
0.00059
0.00034
0.0017
-3846.15
1704.216
48
0.998
0.999
0.999
0.998
0.00073
0.00067
0.00043
0.0018
-3808.54
1489.603
13
0.998
0.999
0.999
0.998
0.00076
0.00043
0.0005
0.0017
-3846.15
2310.696
14
1-5-10-1
1-10-5-1
1-5-5-1
MSEall
1-10-10-1
R2tr
1-15-10-1
Net
Fitness Itera.
This table above represent the sample of finding the best architecture of RNN models and in Table (A) in appendices can be seen completely in detail.
70
Chapter Three: Application Part
Table (3-4) finding the best architecture of RNN model for data under consideration. 1st Hidden layer
2ndHidden layer
Output layer
A.F
A.F
A.F
1-5-1
Tansig
-----------
Purelin
0.99842
1-6-1
Tansig
-----------
Purelin
1-7-1
Tansig
-----------
1-8-1
Tansig
1-9-1
Net.
R2
MSE
Fitness
AIC
0.0016
2628.95
-4204.04
0.99857
0.0018
2437.538 -4120.54
Purelin
0.99845
0.0015
2872.49
-----------
Purelin
0.99917
0.0030
3335.223 -3772.42
Tansig
-----------
Purelin
0.99844
0.0016
2949.591 -4180.04
1-10-1
Tansig
-----------
Purelin
0.99911
0.0029
2848.273 -3782.72
1-11-1
Tansig
-----------
Purelin
0.9991
0.0028
2958.142 -3799.81
1-12-1
Tansig
-----------
Purelin
0.99849
0.0016
3095.879 -4162.04
1-13-1
Tansig
-----------
Purelin
0.99839
0.0016
2878.112 -4156.04
1-14-1
Tansig
-----------
Purelin
0.99843
0.0016
2364.793 -4150.04
1-15-1
Tansig
-----------
Purelin
0.99913
0.0030
2185.888 -3730.42
1-20-1
Tansig
-----------
Purelin
0.99845
0.0016
3696.584 -4114.04
1-25-1
Tansig
-----------
Purelin
0.99834
0.0015
3125.098 -4126.51
1-5-1
Logsig
-----------
Purelin
0.99862
0.0018
2549.07
1-6-1
Logsig
-----------
Purelin
0.99919
0.0030
2876.953 -3784.42
1-7-1
Logsig
-----------
Purelin
0.99837
0.0028
2561.738
model
71
-3727.81
-4126.54
-3705.81
Chapter Three: Application Part
Table (3-4) continues. 1-8-1
Logsig
-----------
Purelin
0.99845
0.0016
2859.757 -4186.04
1-9-1
Logsig
-----------
Purelin
0.99841
0.0015
3068.143 -4222.51
1-10-1
Logsig
-----------
Purelin
0.99912
0.0015
2891.845 -4216.51
1-11-1
Logsig
-----------
Purelin
0.99837
0.0015
2633.242 -4210.51
1-12-1
Logsig
-----------
Purelin
0.99915
0.0030
2607.97
1-13-1
Logsig
-----------
Purelin
0.99848
0.0016
2159.221 -4156.04
1-14-1
Logsig
-----------
Purelin
0.99911
0.0028
3582.303 -3781.81
1-15-1
Logsig
-----------
Purelin
0.99845
0.0017
2625.706 -4104.15
1-20-1
Logsig
-----------
Purelin
0.99846
0.0016
3289.365 -4114.04
1-25-1
Logsig
-----------
Purelin
0.99849
0.0016
3264.134 -4084.04
1-5-1
Logsig
-----------
Logsig
0.84993
0.4960
6.798097 -429.376
1-6-1
Logsig
-----------
Logsig
0.86145
0.4957
5.167959 -423.774
1-7-1
Logsig
-----------
Logsig
0.84735
0.5516
6.246096 -347.465
1-8-1
Logsig
-----------
Logsig
0.84993
0.5007
5.467469
1-9-1
Logsig
-----------
Logsig
0.85367
0.4857
6.257822 -419.184
1-10-1
Logsig
-----------
Logsig
0.84711
0.5139
5.678592 -376.048
1-11-1
Logsig
-----------
Logsig
0.85755
0.4670
6.21118
1-12-1
Logsig
-----------
Logsig
0.84938
0.4720
6.381621 -420.011
1-13-1
Logsig
-----------
Logsig
0.85262
0.4745
5.885815 -410.535
1-14-1
Logsig
-----------
Logsig
0.8473
0.5084
5.662514 -359.128
1-15-1
Logsig
-----------
Logsig
0.84886
0.5415
5.291005 -311.625
1-20-1
Logsig
-----------
Logsig
0.84551
0.5172
6.397953 -311.836
1-25-1
Logsig
-----------
Logsig
0.85481
0.4776
5.827506
1-5-1
Tansig
-----------
Tansig
0.99917
0.0029
2476.658 -3812.72
1-6-1
Tansig
-----------
Tansig
0.9985
0.0017
1953.507 -4158.15
72
-3748.42
-405.17
-433.018
-334.25
Chapter Three: Application Part 1-7-1
Tansig
-----------
Tansig
0.99913
0.0029
2939.361 -3800.72
1-8-1
Tansig
-----------
Tansig
0.99764
0.0026
1826.184 -3866.58
1-9-1
Tansig
-----------
Tansig
0.99917
0.0030
500
-3766.42
1-10-1
Tansig
-----------
Tansig
0.99814
0.0022
2238.188
-3964.5
1-11-1
Tansig
-----------
Tansig
0.999846
0.0016
3282.671 -4168.04
1-12-1
Tansig
-----------
Tansig
0.99679
0.0029
1008.441 -3770.72
1-13-1
Tansig
-----------
Tansig
0.99847
0.0018
1893.366 -4078.54
1-14-1
Tansig
-----------
Tansig
0.99844
0.0016
2023.268 -4150.04
1-15-1
Tansig
-----------
Tansig
0.99842
0.0018
2123.503 -4066.54
1-20-1
Tansig
-----------
Tansig
0.99844
0.0016
2047.167 -4114.04
1-25-1
Tansig
-----------
Tansig
0.99852
0.0018
1622.007 -4006.54
1-5-1
Logsig
-----------
Tansig
0.99907
0.0029
1-6-1
Logsig
-----------
Tansig
0.99846
0.0016
2732.017 -4198.04
1-7-1
Logsig
-----------
Tansig
0.99846
0.0016
2308.509 -4192.04
1-8-1
Logsig
-----------
Tansig
0.99858
0.0018
1826.918 -4108.54
1-9-1
Logsig
-----------
Tansig
0.9984
0.0015
2640.055 -4222.51
1-10-1
Logsig
-----------
Tansig
0.99836
0.0016
2802.298 -4174.04
1-11-1
Logsig
-----------
Tansig
0.99911
0.0029
2169.197 -3776.72
1-12-1
Logsig
-----------
Tansig
0.9992
0.0030
2493.393 -3748.42
1-13-1
Logsig
-----------
Tansig
0.99843
0.0016
3185.018 -4156.04
1-14-1
Logsig
-----------
Tansig
0.99845
0.0015
2857.633 -4192.51
1-15-1
Logsig
-----------
Tansig
0.99917
0.0029
476.1905 -3752.72
1-20-1
Logsig
-----------
Tansig
0.99924
0.0031
1680.814 -3678.84
1-25-1
Logsig
-----------
Tansig
0.99918
0.0030
2012.761 -3670.42
1-5-1
Tansig
-----------
Logsig
0.84742
0.5100
5.605381 -411.061
1-6-1
Tansig
-----------
Logsig
0.85831
0.4762
6.548788 -450.182
1-7-1
Tansig
-----------
Logsig
0.85185
0.5005
7.220217 -411.433
500
73
-3812.72
Chapter Three: Application Part 1-8-1
Tansig
-----------
Logsig
0.85332
0.5040
5.640158 -400.848
1-9-1
Tansig
-----------
Logsig
0.85282
0.4988
6.85401
1-10-1
Tansig
-----------
Logsig
0.85299
0.5049
5.344735 -387.674
1-11-1
Tansig
-----------
Logsig
0.85682
0.4668
6.313131
1-12-1
Tansig
-----------
Logsig
0.84329
0.5101
6.447453 -368.932
1-13-1
Tansig
-----------
Logsig
0.84668
0.5312
5.405405 -336.262
1-14-1
Tansig
-----------
Logsig
0.8516
0.4785
6.02047
1-15-1
Tansig
-----------
Logsig
0.85959
0.4985
5.608525 -366.068
1-20-1
Tansig
-----------
Logsig
0.8532
0.4901
6.807352
1-25-1
Tansig
-----------
Logsig
0.84687
0.5064
5.841121 -295.722
1-5-5-1
Logsig
Logsig
Purelin
0.99848
0.0016
2610.489
-4144.04
1-10-5-1
Logsig
Logsig
Purelin
0.99915
0.0030
2862.705
-3660.42
1-5-10-1
Logsig
Logsig
Purelin
0.99836
0.0015
3644.182
-4116.51
1-10-10-1
Logsig
Logsig
Purelin
0.9991
0.0029
3558.212
-3562.72
1-15-10-1
Logsig
Logsig
Purelin
0.99849
0.0017
3357.846
-3794.15
1-10-15-1
Logsig
Logsig
Purelin
0.99854
0.0017
2564.892
-3794.15
1-15-15-1
Logsig
Logsig
Purelin
0.99852
0.0016
2231.794
-3664.04
1-5-5-1
Tansig
Tansig
Purelin
0.99843
0.0016
2844.707
-4144.04
1-10-5-1
Tansig
Tansig
Purelin
0.99843
0.0016
2299.221
-4074.04
1-5-10-1
Tansig
Tansig
Purelin
0.9991
0.0015
4176.063 -4234.51
1-10-10-1
Tansig
Tansig
Purelin
0.99845
0.0017
2647.113
-3914.15
1-15-10-1
Tansig
Tansig
Purelin
0.99849
0.0017
2310.696
-3794.15
1-10-15-1
Tansig
Tansig
Purelin
0.99837
0.0018
3779.147
-3876.51
1-15-15-1
Tansig
Tansig
Purelin
0.99852
0.0017
2061.006
-3624.15
1-5-5-1
Logsig
Tansig
Purelin
0.99921
0.0030
2357.712
-3730.42
1-10-5-1
Logsig
Tansig
Purelin
0.99906
0.0030
3033.244
-3660.42
1-5-10-1
Logsig
Tansig
Purelin
0.99841
0.0016
3162.755
-4074.04
74
-401.672
-433.3
-399.011
-347.25
Chapter Three: Application Part 1-10-10-1
Logsig
Tansig
Purelin
0.99911
0.0029
3597.769
-3562.72
1-15-10-1
Logsig
Tansig
Purelin
0.9984
0.0016
3005.982
-3834.04
1-10-15-1
Logsig
Tansig
Purelin
0.9984
0.0017
2471.394
-3794.15
1-15-15-1
Logsig
Tansig
Purelin
0.99851
0.0017
2648.796
-3624.15
1-5-5-1
Tansig
Logsig
Purelin
0.99845
0.0016
2765.257
-4144.04
1-10-5-1
Tansig
Logsig
Purelin
0.99914
0.0029
2063.472
-3682.72
1-5-10-1
Tansig
Logsig
Purelin
0.99853
0.0017
2160.574
-4034.15
1-10-10-1
Tansig
Logsig
Purelin
0.99907
0.0027
3614.676
-3609.74
1-15-10-1
Tansig
Logsig
Purelin
0.99842
0.0016
3050.175
-3834.04
1-10-15-1
Tansig
Logsig
Purelin
0.99908
0.0028
2815.078
-3465.81
1-15-15-1
Tansig
Logsig
Purelin
0.99916
0.0029
2238.789
-3272.72
From the table (3-5) that contain finding the best activation function for the best architectures network (1-5-10-1). In this case the best activation function between layers are (Softmax) between input layer and first hidden layer, (Logsig) between first hidden layer and second hidden layer and (Tansig) between second hidden layer and output layer. Also to choose the best activation function depend on maximum value (R2 = 0.99949) and minimum value (MSE = 0.0016 and AIC = -4086.04).
75
Chapter Three: Application Part
Table (3-5)* represent finding the best activation function for the best architecture network [1-5-10-1]. First hidden activation function
Second hidden activation function
Output activation function
R2
MSE
AIC
Tansig
Tansig
Tansig
0.99854
0.0018
-4008.54
Logsig
Logsig
Logsig
0.85815
0.4836
-328.035
Purelin
Purelin
Purelin
0.99837
0.0017
-4046.15
Purelin
Tansig
Logsig
0.85395
0.4869
-323.56
Purelin
Logsig
Tansig
0.99922
0.0031
-3650.84
Logsig
Purelin
Tansig
0.99909
0.0029
-3694.72
Tansig
Purelin
Logsig
0.85893
0.5010
-304.776
Tansig
Tansig
Softmax
0.85389
0.4839
-327.627
Logsig
Logsig
Softmax
0.84953
0.4732
-342.34
Softmax
Logsig
Tansig
0.99949
0.0016
-4086.04
Softmax
Tansig
Logsig
0.85452
0.5113
-291.386
Softmax
Purelin
Purelin
0.99903
0.0028
-3717.81
Purelin
Softmax
Purelin
0.99842
0.0017
-4046.15
Purelin
Purelin
Softmax
0.82816
0.4877
-322.48
Logsig
Purelin
Softmax
0.86144
0.5023
-303.071
Tansig
Purelin
Softmax
0.84559
0.5111
-291.643
Tansig
Elliotsig
Logsig
0.85714
0.4968
-310.316
Logsig
Elliotsig
Tansig
0.7272
3.2331
-922.1245
Elliotsig
Elliotsig
Elliotsig
0.99839
0.0017
-4046.15
Tansig
Logsig
Elliotsig
0.99841
0.0019
-3972.96
Logsig
Tansig
Elliotsig
0.99906
0.0028
-3717.81
Determining the best activation function in suggested model (1-5-10-1)
76
Chapter Three: Application Part
Regression Plot: The regression plot in figure (3 (3-4) consists of (R2 training, R2testing, R2validation and R2 for all data) with the model output for each cases.
Figure (3-4) shown that plot regression of (training, training, testing, validation and all data). From the figure (3-4) 4) that contains the regression plot of (training, testing, validation, and all data) shows the best performance of the detected recurrent neural network (1-5-10-1) 1) model, also it represents the optimal architecture that represents presents all sets as describ describee above this regression plot, also tells as how can the model that we suggested iis the best one among several trails for finding the best architecture moreover also the regression plot tells us that the errors that may be produced from this (RNN) is approximately distributed normally, also their weights for all layers in suggested netw network.
77
Chapter Three: Application Part
3-2-33 Results: Prediction and Forecasting steps The table (3-6) showss the result Recurrent Neural Network for time series prediction where the (R2) and (MSE) for the model (1 (1-5-10-1) 1) are the below: R2= 0.9991, MSE= 0.0015 0.0015.
Figure (3- 5) represent the error histogram in training. training
From the figure above it represent represents as that the error produced after comparing the actual and the output of the suggested network is distributed normally that makes the result more efficient than any other weight distribution. This make as to decide that actually the network in general if the errors are distributed normally it’s really comes from a normal weight set that estimated for the suggested network this also make as to say that the model (1 (1-5--10-1) is more generalized than the others that not distributed normally, also can say the errors if they are random then that distributed normally. The random of error that is necessary for fitting any model.
78
Chapter Three: Application Part Table (3-6) represents the result of applying the (RNN) model (1-5-10-1) for (141Obs) No. Actual Data*
Prediction
No. Actual Data*
Prediction
No. Actual Data*
Prediction
1
-0.77366
-0.7661
26
-0.56935
-0.5581
51
0.158641
0.1746
2
-0.93769
-0.9344
27
-0.48318
-0.4703
52
-0.01266
0.0066
3
-0.74725
-0.7392
28
-0.77516
-0.7677
53
-0.82219
-0.8157
4
-0.58468
-0.5737
29
-0.02902
-0.0097
54
0.873516
0.892
5
0.582171
0.5848
30
-0.5513
-0.5397
55
0.035943
0.0547
6
-0.26887
-0.2516
31
-0.0977
-0.0783
56
-0.01613
0.0032
7
-0.76731
-0.7597
32
-0.80967
-0.8029
57
0.730319
0.7367
8
-0.44057
-0.4268
33
0.681677
0.6859
58
-0.12256
-0.1033
9
-0.54721
-0.5355
34
0.727859
0.7341
59
0.887074
0.9071
10
0.256599
0.269
35
-0.26466
-0.2473
60
0.09349
0.1112
11
0.363957
0.3721
36
0.768738
0.7775
61
0.116401
0.1335
12
-0.10777
-0.0884
37
-0.85755
-0.8519
62
0.097847
0.1154
13
0.44942
0.4544
38
-0.25909
-0.2416
63
-0.63548
-0.6254
14
-0.40331
-0.3888
39
0.574182
0.5768
64
-0.81128
-0.8046
15
0.593851
0.5965
40
-0.75201
-0.7441
65
-0.14145
-0.1223
16
-0.35806
-0.3426
41
-0.15024
-0.1312
66
-0.14688
-0.1278
17
-0.25436
-0.2368
42
-0.38228
-0.3673
67
0.675147
0.6792
18
-0.72289
-0.7144
43
0.871025
0.8892
68
-0.3334
-0.3174
19
0.938508
0.965
44
0.304643
0.3152
69
0.003947
0.0231
20
0.524203
0.5274
45
-0.65792
-0.6482
70
0.619957
0.6228
21
0.562688
0.5654
46
0.090049
0.1078
71
0.803923
0.8155
22
-0.34276
-0.327
47
-0.0216
-0.0023
72
-0.59897
-0.5883
23
-0.59117
-0.5803
48
-0.21095
-0.1927
73
-0.24983
-0.2322
24
-0.28789
-0.271
49
0.02315
0.0421
74
0.049048
0.0676
25
0.429693
0.4353
50
-0.25813
-0.2407
75
0.145818
0.1621
79
Chapter Three: Application Part
No.
actual Data
Prediction No. Actual Data
Prediction No. Actual Data
76
-0.4001
-0.3855
101
0.368641
0.3765
126
-0.09577
-0.0764
77
0.019136
0.0381
102
-0.71999
-0.7114
127
0.018034
0.037
78
-0.4128
-0.3985
103
0.176466
0.1918
128
0.080125
0.0981
79
-0.54962
-0.538
104
0.476339
0.4805
129
-0.47815
-0.4652
80
-0.70843
-0.6997
105
-0.05046
-0.031
130
-0.8563
-0.8507
81
-0.70815
-0.6994
106
0.056412
0.0748
131
-0.07001
-0.0506
82
0.043498
0.0621
107
1
1.0351
132
0.215313
0.2293
83
-0.02718
-0.0078
108
-0.61553
-0.6051
133
0.164919
0.1807
84
-0.25715
-0.2397
109
-0.62679
-0.6166
134
0.814145
0.8266
85
-0.70076
-0.6918
110
0.866339
0.884
135
-0.09809
-0.0787
86
0.375663
0.3833
111
-0.34089
-0.325
136
-0.72307
-0.7146
87
-0.07533
-0.0559
112
0.514636
0.518
137
-0.4031
-0.3886
88
-0.75653
-0.7487
113
-0.54464
-0.5329
138
-0.86611
-0.8607
89
0.286947
0.2982
114
0.311792
0.322
139
-0.69755
-0.6886
90
-0.80046
-0.7935
115
-0.63504
-0.625
140
0.028908
0.0477
91
-0.1132
-0.0939
116
-0.79252
-0.7854
141
-0.04048
-0.0211
92
-0.13856
-0.1194
117
0.356348
0.3647
93
0.91838
0.9422
118
-0.83204
-0.8258
94
0.657958
0.6615
119
0.50266
0.5062
95
0.956837
0.9858
120
0.373797
0.3815
96
-0.82934
-0.823
121
-0.48583
-0.473
97
0.694498
0.6992
122
-0.08176
-0.0623
98
-0.5742
-0.563
123
0.263073
0.2753
99
0.965315
0.9955
124
-0.11013
-0.0908
100
-0.2699
-0.2527
125
0.736161
0.7429
80
Prediction
Chapter Three: Application Part 1.5 1 0.5 Actual 0 Prediction -0.5 -1 -1.5
Figure (3-6) 6) represents the difference between Actual data and prediction.
From the table (3-6) that represent represents the prediction of demand on electric power energy to show the performance of suggested model (1 (1-5-10-1) 1) that gives as MSE= 0.0015, R2= 0.9991 0.9991for for over all data (training, testing, and validation set) comparing this result by the others it gives as the best among the epochs used in our data. From the figure (3 (3-6) that shown the difference between the actual data (demand on electric power energy this data normalized by the equation (3-1)) (3 and prediction for (141 days) in the validation set set.
81
Chapter Three: Application Part
Weight and Bias Between layers for the suggested model (1-5-10-1) IW{1,1}: the input weight matrix by [5∗1]
0.3491
ۍ0.8992 ې ێ ۑ ێ−0.0164ۑ ێ−0.6563ۑ ۏ0.1305 ے
LW {2, 1}: the weight between first hidden layers and second hidden layers matrix [10∗5]. 0.4077
ۍ0.1777 ێ −0.00037 ێ ێ0.1528 ێ−0.4366 ێ−0.4529 ێ0.2791 ێ−0.1923 ێ−0.4880 ۏ0.5558
−0.4388 −0.3840 0.2603 0.4606 −0.4702 −0.0703 0.1095 −0.3348ې 0.0775 0.4233 0.2345 −0.4236ۑۑ −0.0689 0.0746 −0.0734 0.2197 ۑ −0.4988 0.2434 0.1546 −0.4264ۑ −0.4442 0.4744 −0.3189 −0.2965ۑ −0.2901 0.0979 0.4177 0.1188 ۑ −0.4067 0.0034 −0.2710 0.2288 ۑ 0.1467 0.0356 −0.1616 0.4396 ۑ −0.0310 0.1396 −0.2582 0.1348 ے
LW {3, 2}: the weight between second hidden layers and output layer matrix [10∗1]. Transpose ሾ1.0212 −0.1494 −0.3969 0.7214 0.1839 0.8332 −0.7677 0.6509 −0.5051 0.7825ሿ
b {1}:Bias first hidden layer matrix [1∗5]. ሾ−1.7829
−0.9650
0.4342
−0.5143
−1.5869ሿ
b {2}:Bias second hidden layer matrix [1∗10]. ሾ−1.5196 1.3136 −0.8545 0.7075 −0.1454 −0.2896 0.5243 −0.6976 −1.1426 1.5371ሿ
b {3}:Bias output layer matrix [1∗1]. ሾ0.2880ሿ
82
Chapter Three: Application Part
Figure (3 (3-7) represents the weight distribution From figure above these weights between layers in suggesting network are distributed normally. This can be compared with the error of suggested model (1-5-10-1) 1) that shown in figure (3 (3-5) 5) it tells us that the normality of the weights attains to the normality the error of errors and vi vice versa. From the table (3-8)) and figure (3 (3-8)) firstly we make our prediction and using the predicted model (1-5--10-1) 1) to forecast the demand of power electric energy and we get et results as shown in column (2 (2)) for forecast values using suggested model to make sure that the fit model has a good performance and more generalization we make a comparison after waiting (2 months) till to get the actual data after making comparison the difference (error) between forecast and actual values for this time period daily (60 days) fortunately we get the difference between them as shown in column (D) from table (3-9)) are minimum as possible, le, moreover see the figure (3 (3-8)) the behaviors of these two variables are equally likely. The table (3-7)) also forecasting for two months. But in this process actual data and forecasting values by using equation (3-1) normalization.
83
Chapter Three: Application Part
Table (3-7) represents the result of applying the (RNN) model (1-5-10-1) for (60) observation after (940) observation.
No.
Actual Data*
Forecast Data
No.
Actual Data*
Forecast Data
No.
Actual Data*
Forecast Data
1
0.8709
0.8274
21
0.3884
0.3234
41
-0.235
-0.2628
2
0.9001
0.8598
22
0.3971
0.3319
42
-0.334
-0.3552
3
0.9446
0.9094
23
0.4974
0.4315
43
-0.167
-0.1996
4
0.9994
0.9706
24
0.3777
0.3129
44
-0.042
-0.0833
5
1
0.9713
25
0.3015
0.2392
45
-0.052
-0.0927
6
0.955
0.921
26
0.1884
0.1317
46
0.0235
-0.0224
7
0.6705
0.6101
27
-0.037
-0.0788
47
-0.07
-0.1091
8
0.7312
0.6749
28
-0.319
-0.3417
48
-0.183
-0.2147
9
0.7213
0.6642
29
-0.129
-0.1638
49
-0.45
-0.4645
10
0.749
0.694
30
0.0482
0.0005
50
-0.48
-0.4934
11
0.8436
0.7973
31
-0.163
-0.1961
51
-0.54
-0.5501
12
0.9671
0.9345
32
-0.268
-0.2934
52
-0.471
-0.4849
13
0.951
0.9165
33
-0.292
-0.3165
53
-0.465
-0.4786
14
0.7881
0.7365
34
-0.197
-0.227
54
-0.39
-0.4083
15
0.8359
0.7889
35
-0.257
-0.2834
55
-0.815
-0.8113
16
0.8713
0.8279
36
-0.028
-0.0704
56
-0.951
-0.9401
17
0.7894
0.7379
37
-0.037
-0.0785
57
-1
-0.9855
18
0.6952
0.6364
38
-0.806
-0.8026
58
-0.989
-0.9756
19
0.6491
0.5875
39
-0.747
-0.7471
59
-0.794
-0.7916
20
0.6071
0.5437
40
-0.388
-0.406
60
-0.708
-0.7096
This table above represents the forecasting for 2 months after (940) data point. Actual Data*=data after normalization by the eq. (3-1).
84
Chapter Three: Application Part
Table (3-8) represents the result of forecasting for two months after (940) observation.
No.
Actual Data
Forecast Data
No.
Actual Data
Forecast Data
No.
Actual Data
Forecast Data
1
120.304
118.418
21
108.324
105.792
41
92.8419
92.0488
2
121.029
119.152
22
108.539
105.992
42
90.3906
89.5337
3
122.135
120.225
23
111.03
108.414
43
94.5274
93.7207
4
123.496
121.459
24
108.058
105.547
44
97.6351
96.646
5
123.51
121.472
25
106.166
103.865
45
97.3838
96.4181
6
122.393
120.467
26
103.356
101.504
46
99.2631
98.0846
7
115.328
113.012
27
97.754
96.7533
47
96.9451
96.0165
8
116.836
114.686
28
90.7489
89.9059
48
94.1249
93.3263
9
116.589
114.411
29
95.4818
94.6427
49
87.5087
86.56
10
117.277
115.173
30
99.8745
98.6093
50
86.7512
85.8056
11
119.627
117.714
31
94.6186
93.8096
51
85.266
84.3749
12
122.693
120.743
32
92.0282
91.2235
52
86.973
86.0247
13
122.292
120.372
33
91.416
90.5958
53
87.1384
86.1891
14
118.249
116.239
34
93.7967
93.0024
54
88.9893
88.0756
15
119.436
117.513
35
92.2949
91.4953
55
78.4418
78.0176
16
120.314
118.428
36
97.9784
96.9547
56
75.0535
74.7161
17
118.282
116.275
37
97.7628
96.7612
57
73.8467
73.5513
18
115.943
113.694
38
78.6712
78.2385
58
74.1119
73.8055
19
114.796
112.424
39
80.1216
79.6187
59
78.9583
78.514
20
113.754
111.283
40
89.0491
88.1376
60
81.1006
80.5333
85
Chapter Three: Application Part
Figure (3-8)) represents the actual data and forecast data for two months. Table (3-9)) represents the Differences between actual data and forecasting for two months. (D= Actual ctual data – Forecast data). No.
D
No.
D
No.
D
No.
D
1
-1.886
16
-1.886
31
-0.809
46
-1.1785
2
-1.877
17
-2.007
32
-0.8047
47
-0.9286
3
-1.91
18
-2.249
33
-0.8202
48
-0.7986
4
-2.037
19
-2.372
34
-0.7943
49
-0.9487
5
-2.038
20
-2.471
35
-0.7996
50
-0.9456
6
-1.926
21
-2.532
36
-1.0237
51
-0.8911
7
-2.316
22
-2.547
37
-1.0016
52
-0.9483
8
-2.15
23
-2.616
38
-0.4327
53
-0.9493
9
-2.178
24
-2.511
39
-0.5029
54
-0.9137
10
-2.104
25
-2.301
40
-0.9115
55
-0.4242
11
-1.913
26
-1.852
41
-0.7931
56
-0.3374
12
-1.95
27
-1.0007
42
-0.8569
57
-0.2954
13
-1.92
28
-0.843
43
-0.8067
58
-0.3064
14
-2.01
29
-0.8391
44
-0.9891
59
-0.4443
15
-1.923
30
-1.2652
45
-0.9657
60
-0.5673
86
Chapter Three: Application Part
3-3 Results and Discussions: In general the results in this chapter showed the points below. 1. At this stage to determine the best architecture recurrent neural network model on the road to some measures such as (R2, MSE, Fitness model and AIC) depends on where the lowest value of (AIC and MSE) and largest value of (R2 and Fitness model). Table (3-3) represents the best architecture of RNN for data under consideration and testing for several kind of activation function between layers after determined the suggested Recurrent neural network model show the table (3-5) represent finding the best activation function for the best architecture network. 2. In this step during training suggested the Recurrent Neural Network, the data would be analyzed and change weights among nodes to reflect dependencies and patterns. In this section we made use of training algorithm. Then we chose the best algorithm named by (LevenbergMarquardt) which is explained in figure (3-2) that shows the best training state. It is clear that the best efficiency is occurred in repetition at epoch (12). The learning function of learning data is shown in repetition at epoch (12) in Fig (3-2). 3. Figure (3-3) is the diagram of learning errors, assessment errors and test errors and determining the best training performance with the best validation performance for (RNN) located at epoch (6) because the minimum global located in epoch (6) by amount (0.00028243). 4. Figure (3-5) represent how distributed the error in each part of data (training set, testing set, and validation set) data and determine the errors by using the difference between targets and outputs.
87
Chapter Three: Application Part
5. Table (3-7) represents the result of applying the (RNN) model (1-5-10-1) forecasting for (60) days for power energy after (940) days and this table suggested represent the forecasting in power energy demand for two months after (940) days. The data is used normalization by equation (3-1). 6. Table (3-8) represents the result of applying the (RNN) model (1-5-10-1) forecasting for (60) days for power energy demand after (940) days and this table suggested represent the forecasting in power energy demand for two months after (940) days. 7. After determined forecasting for (60) days for power energy demand after (940) days Figure (3-8) represents the actual data and forecast data for two months and Table (3-9) represents the Differences between actual data and forecasting for two months. (D= actual data – Forecast data). 8. During this study the time series data for power electric the statistical results tells that this type of the data under consideration is a chaotic or temporal or both that can’t be treated and represented by linear models or some type of non-linear models, the model (FFNN), regression models, (NARX) model can’t give a good performance about both (prediction and forecasting) as RNN can do it.
88
Chapter Three: Application Part
9. The predicted model RNN (1-5-10-1) and Its results gives us an idea that the power energy system can’t be expanded in their usages because during the comparison between the load power energy and the demand is a very small error of prediction that we can see it in table (3-9), then we can recommend that if the governorate could not expand and develop the system then it can’t be able to provide a new or some new service and productivity institutions factories because in this time tell know there is a balance can be seen clearly between electric power energy consumption and the actual power energy in use for sulaimani.
89
Chapter Four: Conclusions and Recommendations
4-1 Conclusions: As result of practical part, the following are the main conclusions: 1. The study found the best model of network for data under consideration through (RNN) model is (1-5-10-1) where (1) nodes for input layer, (5) nodes for first hidden layer, (10) nodes for second hidden layer and (1) nodes for output layer with the activation functions between layers are [Tansig1 Tansig2 Purelineoutput], as shown the table (3-2) and the figure (3-1) represents the best architecture of (RNN) model. The suggested recurrent neural network (1-5-10-1) as required for detecting pattern of the data has a performance scale withR2 = 0.9991, MSE = 0.0015, Fitness model = 4176.063 and AIC = -4234.51 as shown the table (3-3) and table (3-4). 2. Figure (3-4) the plot of regression consists of (R2 training, R2testing, R2validation and R2 all data) with the model output for each cases and shows the best performance of the detected recurrent neural network (1-5-10-1) model. Also the regression plot tells us that the error that may be produced from this (RNN) is approximately distributed normally. 3. Table (3-6) represents the result of applying the (RNN) model (1-5-10 1) for (141) observation in (validation set) for electric power energy on demand and show the result Recurrent Neural Network for time series prediction for validation set in electric power energy demand and figure (3-6) the difference between actual data and prediction where the (R2) and (MSE) for the model (1-5-10-1) are the R2= 0.9991, MSE= 0.0015.
90
Chapter Four: Conclusions and Recommendations
4. The estimated model RNN (1-5-10-1) is the best network can be used as a predicted model in practice for electric power energy in sulaimani, also for the quantity of demand of it, so this model can used as a control cart model in order to control any expansion occurred between load energy and the demand, also to watch the performance of the act for electric power establishment in sulaimani governorate that holds this responsibility to make balancing between load and demand on electrical energy available. 5. It’s appear researcher in application part for this study and fitting a prediction model RNN (1-5-10-1) and used it for forecasting to nearly two months daily (60 days) and comparing these forecast values with actual data for the same time period is a guide that the estimated RNN above is as perfect as possible to make us saying that this model is un optimum so it can be used to forecast the quantity of demand on electric power energy to know how much the sulaimani governorate needs of this energy in the future or nearly the near future this make’s the governorate an ability to treat and covering the electric power energy demand for sulaimani.
91
Chapter Four: Conclusions and Recommendations
4-2 Recommendations: After the researcher was finished this study (Using Recurrent Neural Network for Time Series Forecasting of Electric Demand in Sulaimani), also after a deep study for it and resolving the results and there analysis that made, also the study produced some assignments can be formulated as a conclusions for whom it may concerned and the scientific center to take them to apply due to the facilities available. The important points of these recommendations are: 1. The researcher recommended to use this model RNN (1-5-10-1) as a distribution system for electric power energy and the demand to make balancing that recommended the produced power energy and the demand to make an optimal exploitations that helps the sulaimani governorate to save as possible as it can for electric power energy that may lose clearly energy consumption between human consumption and the production establishment consumption. 2. The researcher recommended also continuing about studies for electric power
energy
in
several
spaces
(economic,
technical,
scientific
researches……, etc.). This is for the electric power energy is a goods having a mass its importance for our life then the sulaimani governorate must take the results of these studies and working to accomplish them. 3. As a new line in neural networks and non-linear regression models the researcher recommended are scientific study for this type of data by mixing RNN with non-linear regression models such as the linear regression model be a substitution to the activation function.
92
References
Thesis: 1. Ahmed, Nawzad, M, (2007), “Forecasting water Inflow and Electric Power Generation in Darbandi-khan Station By Using FFNN, and Adaptive Method In Time series”, Doctorate Thesis, Statistics Dept., Administration and economics college, Sulaimani, Iraq. 2. Ahmed, Farhad, A, (2014), “Using FFNN and NARX models to Forecast daily temperature in Sulaimani”, MSc Thesis, Statistics Dept., Administration and economics college, Sulaimani, Iraq. 3. Alex Graves, (2008), “Supervised Sequence Labelling with Recurrent Neural Networks”, Technische Universita t Mu nchen Fakulta t fu r Informatik, PhD. Thesis. 4. Gilde, C., (1996), “Time series analysis and prediction using recurrent gated experts”, M.Sc. thesis, University of Sko vde. 5. Gupta, Charu, (2006), “Implementation of Back Propagation Algorithm (of neural networks) In VHDL”, Master thesis, Department of Electronics and Communication Engineering, Thapar Institute of Engineering & Technology, Patiala, India. 6. Nielsen. Fiona 4i, (2001), “Neural Networks – algorithms and applications”, Supervisor: Geert Rasmussen, Niels Brock Business College. 7. Sutskever,
Ilya,
(2013),
“TRAINING
RECURRENT
NEURAL
NETWORKS”, a thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy Graduate Department of Computer Science University of Toronto. 8. Yu. Hao, (2010), “Levenberg–Marquardt Training”, Bogdan M. Wilamowski, Auburn University.
93
References
Books: 9. Anderson, Dave and McNeill, George, (1992), “ARTIFICIAL NEURAL NETWORKS TECHNOLOG”, Kaman Sciences Corporation, 258 Genesse Street, Utica, New York 13502-4627. 10.Azoff, E. M., (1994), “Neural Network Time series forecasting of financial markets”, Chi Chester, England, john Wiley & sons. 11.Graupe, Daniel, (2007), “PRINCIPLES OF ARTIFICIAL NEURAL NETWORKS”, Second Edition, University of lllinois, Chicago, USA. 12.Greshenfeld, N.A. and Weigned, A. S., (1993), “The future of Time Series: Forecasting the Future And Understanding the Past”, lCore funding for the Santa Fe Institute is provided by the John D. and Catherine T. MacArthur Foundation, the National Science Foundation, grant PHY8714918, and the U.S. Department of Energy, grant ER-FG05-88ER25054. 13.Greshenfeld, N.A. and Weigned, A. S., (1994), “The future of Time Series: Learning and Understanding”, Greshenfeld, N.A. and Weigned, A. S., editors. Time Series Prediction, Reading, Massachusetts, Addison-Wesley publishing Company, pp.1-70. 14.Johan A. K. SuykensJony Van Gestel, Jos De Brabanter, Bart De Moor and JoosVandewalleK. U. Leuven, Belgium(first published 2002 Reprinted 2005), “Least Squares Support Vector Machines”, British Library Cataloguing-in-Publication Data.
94
References
15. Kantz. Holger& Schreiber.
Thomas, (2004), “Nonlinear Time Series
Analysis”, Second Edition, Published (2004), Printed in the United Kingdom at the University Press, Cambridge. 16. Kmenta, Jan (1986). “Elements of Econometrics”, 2nd edition. New York: Macmillan. pp. 240–243. ISBN 0-02-365070-2. 17. L.R,Medsker,
and
L.C,Jain,
(2001)
“RECURRENT
NEURAL
NETWORKS design and application”, firstly, Departments of Physics and Computer
Science and
Information Systems
American
University
Washington, D.C. and secondly, Knowledge-Based Intelligent Engineering Systems Centre Faculty of Information Technology Director/Founder, KES University of South Australia, Adelaide The Mawson Lakes, SA Australia. 18. Lehmann, E. L.; Casella, George (1998). “Theory of Point Estimation”, 2nd edition. New York: Springer. ISBN 0-387-98502-6. MR 1639875. 19. Makridakis. Spyros, Steven C.Wheelwright and Rob Hyndman, (1998), “FORECASTING Methods and Application”, third Edition, (1998), Johan Wiely and Sons, Inc, United States of America. 20.Menhaj. Mohammed. B, (2013), “Computational Intelligence (Val.1) FUNDAMENTAL of NEURAL NETWORKS”, Eighth Edition, Amir Kabir Publisher, Iran. 21. MATLAB textbook help, in Matlab Software R2014a. 22.Pyle, D., (1999), “Data preparation for data mining”, San Francisco, Morgan Kaufmann Publishers, Inc. 23. Robert H. Shumway & David S. Stoffer (2011), “Time Series Analysis and Its Applications”, 3rd additions, Department of Statistics University of California
Davis,
California
USA,
published
in:
www.springer.com/series/417
95
References
24.Suzuki,
Kenji,
(2011),
METHODOLOGICAL
“ARTIFICIAL ADVANCES
NEURAL AND
NETWORKS BIOMEDICAL
APPLICATION”, Published by InTech, JanezaTrdine 9, 51000 Rijeka, Croatia. 25.Thomas, H, (1998), “Neural Network: A Tutorial”, Vasteras University computing Science Dept. Sweden. 26.WILLIAM
C.
CAVE
(2014)
“PREDICTION
THEORY
FOR
CONTROL SYSTEMS”, © COPYRIGHT 1989 THROUGH 2014. 27.Yi, Zhangl& Tan, KK, (2004), “Convergence Analysis of Recurrent Neural Networks”, School of Computer Science and Engineering University of Electronic Science and Technology of China Chengdu 610054 People's Republic of China, Department of Electrical and Computer Engineering The National University of Singapore 4 Engineering Drive 3117576 Singapore. 9
Papers: 28.A.Blanco, M.Delgado, M.C.Pegalajar, (2001), “A real-coded genetic algorithm for training recurrent neural networks”, Department of Computer Science and Artificial Intelligence, E.T.S.I. InformaÂtica, University of Granada, Avenida de AndalucõÂa, 18071 Granada, Spain. 29.Amin-Naseri & RostamiTabar (2008), “Neural network approach to lumpy demand forecasting for spare parts in process industries”, Published in: Computer and Communication Engineering, 2008. ICCCE 2008.
96
References
30.Aymen Chaouachi and Ken Nagasaka (2010), “Multi objective Intelligent Energy
Management
for
a
Micro
grid”,
IEEE TRANSACTIONS ON INDUSTRIAL ELECTRONICS, VOL. 60, N O.4. 31.Bigus, P. J., (1996), “Data mining with neural networks”, New York, McGraw-Hill, and Publisher:Mcgraw-Hill (Tx) (May 20, 1996). 32.Christoffer Brax (2010), “Recurrent neural networks for time-series prediction”, Department of Computer Science, University of Skövde, Box 408, S-54128 Skövde, SWEDEN. 33.C. Lee Giles, Steve Lawrence, A. C. Tsoi, (2001), “Noisy Time Series Prediction using a Recurrent Neural Network and Grammatical Inference”, Machine Learning, Volume 44, Number 1/2, July/August, pp. 161–183. 34.CARRIE KNERR, B.S, (2004), “TIME SERIES PREDICTION USING NEURAL NETWORKS”, A THESIS IN COMPUTER SCIENCE, Submitted to the Graduate Faculty of Texas Tech University in Partial Fulfillment of the Requirements for the Degree of MASTER OF SCIENCE. 35.C.Feng, R.Plamondon, and C.O’Reilly, (2010), “On Some Necessary and Sufficient Conditions for a Recurrent Neural Network Model With Time Delays to Generate Oscillations”, published in: IEEE Transactions on Neural Networks (Volume:21 , Issue: 8 ). 36.Debes. Klaus, Alexander Koenig, Horst-Michael Gross, (2005), “Transfer Function in Artificial Neural Networks A Simulation-Based Tutorial”, Technical University Ilmenau, Germany. 37.Erik, Hulthen, (2004), “Improving Time Series Prediction using Recurrent Neural Networks and Evolutionary Algorithms”, Division of
97
References
Mechatronics Department of Machine and Vehicle Systems Chalmers University of Technology SE-412 96 Goteborg Sweden. 38.Fallon, A. and Spada, C., (1997), “Detection and accommodation of Outliers in Normally Distributed Data Set”, Virginia Tech. Available from:http://www.cee.vt.edu/envior2/smprimer/outlier/outlier.html [Accessed 14/08/00]. 39.FELIX GERS, (2001), “Long Short-Term Memory in Recurrent Neural Networks”, Diplom in Physik, Universita t Hannover, Deutschland de nationalité allemand. 40.Herrn, Anton and Maximilian Schafer, (2008), “Reinforcement Learning with Recurrent Neural Networks”, Erlangung der Doktorwu rde der
Universita t Osnabru ck – Fachbereich Mathematik/Informatik. 41.J.Xu, Y.-Y.Cao, Y.Sun, and J.Tang, (2008), “Absolute Exponential Stability of Recurrent Neural Networks with Generalized Activation Function”, IEEE Trans Neural Netw. 2008 Jun; 19(6):1075-89. doi: 10.1109/TNN.2007.2000060. 42.Jerome Connor, Les E. Atlas, and Douglas R. Martin (1992),“Recurrent Networks
and
NARMA
Modeling”,
Interactive
Systems
Design
Laboratory,Dept. of Electrical Engineering and Dept. of Statistics, University of Washington Seattle, Washington 98195. 43. K. Levenberg. “A Method for the Solution of Certain Non-linear Problems in Least Squares”. Quarterly of Applied Mathematics, 2(2):164– 168, Jul. 1944. 44.Karol Kuna, (2015), “Time Series Prediction Using Neural Networks”, MASARYK UNIVERSITY, FACULTY OF INFORMATICS.
98
References
45.Lipton, Zachary C & Berkowitz, John &Elkan, Charles, (2015), “A Critical Review of Recurrent Neural Networks for Sequence Learning”, University of California, San Diego. 46.Martens, James &Sutskever, Ilya, (2011), “Learning Recurrent Neural Networks with Hessian-Free Optimization”, Appearing in Proceedings of the 28th International Conference on Machine Learning, Bellevue,WA, USA, 2011. 47.Miyoshi, H.-F. Yanai, M.Okada, (2004), “Associative memory by recurrent neural networks with delay elements”, Neural Networks 17 (2004) 55–63. 48.Mohammad Y. El-Sharkh and Mohammad Anwar Rahman, “Forecasting Electricity Demand Using Dynamic Artificial Neural Network Model”, Proceedings of the 2012 International Conference on Industrial Engineering and Operations Management Istanbul, Turkey. 49.Manuel O.Mellado, Angel K.Morales and Jose De Jesus Acosta Flowers, (2013), “number of hidden neurons in a RNN estimation”, International Journal of Scientific & Engineering Research (IJSER), January 2013 ISSN 2229-5518. 50.P.Tino, C.Schittenkopf and G.Dorffner in (2001), “Financial Volatility Trading Using Recurrent Neural Networks”, published in: IEEE Transactions on Neural Networks (Volume: 12, Issue: 4). 51.Panchal, Foram S. and Panchal, Mahesh, (2014), “Review on Methods of Selecting Number of Hidden Nodes in Artificial Neural Network”, Department of Computer Science &Engineering, Kalol Institute of Technology, Gujarat, India published inInternational Journal of Computer Science and Mobile Computing.
99
References
52.Paulo. C, Miguel. R, Jose. N. (2001), “Evolving Time Series Forecasting Neural network Models”, Departamento de Inforḿ tica Universidade do Minho4750-057 Braga, Portugal. 53.S.L.Goh and D.P.Mandic (2003), “Recurrent Neural Networks with trainable amplitude of activation functions”, Imperial College of Science, Technology and Medicine, London SW7 2AZ, UK. 54.Tarik Rashid, B.Q.Huang, M-T.Kechadi and B.Gleeson, (2007), “Autoregressive Recurrent Neural Network Approach for Electri city Load Forecasting”, World Academy of Science, Engineering and Technology International Journal of Electrical, Computer, Energetic, Electronic and Communication Engineering Vol:1, No:10, 2007. 55.W.Lin and G.Chen, (2009), “Large Memory Capacity in Chaotic Artificial Neural Network: A View of the Anti-Integrable Limit”, published in: IEEE Transactions on Neural Networks (Volume:20 , Issue: 8 ). 56.Y.Zhao, H.Gao, J.Lam, and K.Chen, (2009), “Stability Analysis of Discrete-Time Recurrent Neural networks With Stochastic Delay”, published in: IEEE Transactions on Neural Networks (Volume:20 , Issue: 8 ). 57.Y.Alanis, E.N.Sanchez, A.G.Lokianov, and M.A.Perez, (2011), “Real-Time Recurrent Neural State Estimation”, Published in: IEEE Transactions on Neural Networks (Volume: 22, Issue: 3, March 2011).
100
References
Software: 1- MTLAB (R2014a V.8.3.0.532). 2- Microsoft Excel 2013.
101
Appendices
Appendices A: Represents finding the best architecture of RNN model by activation functions [LogsigPurline]. Network
1-5-1
1-6-1
1-7-1
R2tr
R2ts
R2val
R2all
MSEtr
MSEts
MSEval
MSEall
AIC
fitness
MSEModel
Iteration
0.99909
0.99909
0.99922
0.99911
4.5089e-04
0.0020
3.4418e-04
0.0028
-3835.81
500
2.1446e-04
55
0.99914
0.9995
0.99885
0.99915
4.3639e-04
0.0023
2.9575e-04
0.0030
-3790.42
434.7826
1.3397e-04
17
0.99796
0.99927
0.99949
0.99841
8.1312e-04
4.6362e-04
3.1201e-04
0.0016
-4204.04
2156.939
1.4259e-04
13
0.99814
0.99924
0.9988
0.99841
8.0931e-04
4.0447e-04
3.2467e-04
0.0015
-4246.51
2472.371
1.2546e-04
34
0.99924
0.99935
0.99481
0.99862
7.3814e-04
3.9230e-04
6.5728e-04
0.0018
-4126.54
2549.07
1.6730e-04
12
0.99817
0.99918
0.9993
0.9985
7.5121e-04
4.8161e-04
5.4277e-04
0.0018
-4120.54
2076.369
2.0082e-04
14
0.9992
0.99932
0.99903
0.99919
4.1608e-04
3.4759e-04
0.0022
0.0030
-3784.42
2876.953
2.0955e-04
38
0.99825
0.99915
0.99877
0.99846
7.6903e-04
4.8814e-04
3.6567e-04
0.0016
-4198.04
2048.593
1.5997e-04
53
0.99923
0.9993
0.99475
0.99852
7.6575e-04
4.7757e-04
4.4410e-04
0.0017
-4158.15
2093.934
1.4594e-04
30
0.9981
0.99923
0.99927
0.99846
7.8177e-04
4.4913e-04
5.4519e-04
0.0018
-4120.54
2226.527
1.7347e-04
11
0.99818
0.99897
0.99898
0.99842
8.0566e-04
4.1820e-04
2.8297e-04
0.0015
-4234.51
2391.2
1.8789e-04
16
0.99908
0.99426
0.99903
0.99838
8.0926e-04
4.0152e-04
3.1560e-04
0.0015
-4234.51
2490.536
1.7385e-04
15
0.99916
0.9993
0.99944
0.99921
4.0109e-04
0.0021
4.6325e-04
0.0030
-3778.42
476.1905
1.4953e-04
100
0.99918
0.99925
0.99433
0.99846
7.7760e-04
4.4773e-04
3.5560e-04
0.0016
-4192.04
2233.489
1.8407e-04
117
0.99809
0.999
0.99915
0.99837
8.2581e-04
3.9036e-04
2.9231e-04
0.0015
-4234.51
2561.738
1.7598e-04
10
102
Appendices
Network
1-8-1
1-9-1
1-10-1
R2tr
R2ts
R2val
R2all
MSEtr
MSEts
MSEval
MSEall
AIC
fitness
MSEModel
Iteration
0.99815
0.99928
0.99878
0.99843
8.0071e-04
4.8718e-04
3.2297e-04
0.0016
-4186.04
2052.629
1.0807e-04
13
0.99821
0.99896
0.99914
0.99845
8.0364e-04
4.6611e-04
3.1910e-04
0.0016
-4186.04
2145.416
1.2686e-04
9
0.9991
0.99515
0.99913
0.99845
7.8552e-04
3.4968e-04
4.8990e-04
0.0016
-4186.04
2859.757
1.6961e-04
19
0.99819
0.99915
0.99938
0.99851
7.7272e-04
4.3765e-04
4.5333e-04
0.0017
-4146.15
2284.931
3.5033e-04
24
0.99921
0.99488
0.99866
0.99834
8.3230e-04
3.6105e-04
3.6578e-04
0.0016
-4186.04
2769.699
1.6059e-04
10
0.99824
0.99898
0.99926
0.99846
7.8066e-04
3.6752e-04
7.1961e-04
0.0019
-4066.96
2720.94
2.5575e-04
10
0.99794
0.99967
0.99881
0.99839
8.2467e-04
3.8675e-04
3.0382e-04
0.0015
-4222.51
2585.65
1.9422e-04
11
0.99815
0.99915
0.99922
0.99841
8.1443e-04
3.2593e-04
3.6887e-04
0.0015
-4222.51
3068.143
1.5483e-04
16
0.99922
0.9987
0.99886
0.99904
4.8607e-04
0.0020
4.3237e-04
0.0029
-3788.72
500
2.0136e-04
9
0.99803
0.99949
0.99944
0.99846
7.5778e-04
3.9316e-04
5.8647e-04
0.0017
-4140.15
2543.494
1.7616e-04
32
0.99823
0.9992
0.99937
0.99853
7.4674e-04
4.3134e-04
6.2773e-04
0.0018
-4096.54
2318.357
1.5844e-04
37
0.99827
0.99904
0.99899
0.99849
7.7780e-04
5.6759e-04
3.3263e-04
0.0017
-4134.15
1761.835
1.6350e-04
64
0.99835
0.99933
0.99925
0.99861
7.1592e-04
6.2166e-04
4.7045e-04
0.0018
-4096.54
1608.596
1.4529e-04
44
0.99914
0.99917
0.99909
0.99912
7.9591e-04
3.4580e-04
3.8508e-04
0.0015
-4216.51
2891.845
1.3138e-04
11
0.99917
0.99901
0.99916
0.99914
4.3450e-04
0.0020
4.8723e-04
0.0029
-3782.72
500
7.4131e-05
12
103
Appendices
Network
1-11-1
1-12-1
1-13-1
R2tr
R2ts
R2val
R2all
MSEtr
MSEts
MSEval
MSEall
AIC
fitness
MSEModel
Iteration
0.99922
0.99906
0.99881
0.99913
4.4814e-04
4.4228e-04
0.0020
0.0029
-3776.72
2261.011
1.9090e-04
10
0.99917
0.99916
0.99455
0.99854
8.9272e-04
8.2182e-04
6.8269e-04
0.0024
-3901.24
1216.811
2.6722e-04
11
0.99809
0.99863
0.99941
0.99837
8.3287e-04
3.7976e-04
2.9935e-04
0.0015
-4210.51
2633.242
1.9618e-04
12
0.99922
0.99882
0.99931
0.99915
4.2251e-04
0.0021
5.0828e-04
0.0030
-3754.42
476.1905
2.1384e-04
14
0.99924
0.99923
0.99922
0.99924
3.8984e-04
4.3641e-04
0.0022
0.0031
-3732.84
2291.423
2.0617e-04
20
0.99928
0.99856
0.99905
0.99915
4.2266e-04
0.0020
5.1244e-04
0.0029
-3770.72
500
2.1087e-04
68
0.99818
0.99948
0.99897
0.99849
7.7442e-04
4.9786e-04
4.4727e-04
0.0017
-4122.15
2008.597
1.5440e-04
12
0.9992
0.99902
0.99908
0.99915
4.2322e-04
3.8344e-04
0.0022
0.0030
-3748.42
2607.97
1.5974e-04
19
0.99917
0.99932
0.99941
0.99923
4.0495e-04
0.0022
4.1078e-04
0.0030
-3748.42
454.5455
1.4537e-04
23
0.99912
0.99941
0.99494
0.9985
7.8415e-04
3.9651e-04
4.2974e-04
0.0016
-4162.04
2522.004
1.5655e-04
109
0.99825
0.9988
0.99948
0.99851
7.4956e-04
5.1969e-04
6.4186e-04
0.0019
-4042.96
1924.224
1.4782e-04
9
0.99807
0.99947
0.99929
0.99848
7.9252e-04
4.6313e-04
3.3928e-04
0.0016
-4156.04
2159.221
1.7490e-04
68
0.99829
0.99926
0.99939
0.99857
7.4721e-04
4.9925e-04
4.7128e-04
0.0017
-4116.15
2003.005
1.4320e-04
53
0.99921
0.99944
0.99912
0.99922
4.1740e-04
0.0021
4.6987e-04
0.0030
-3742.42
476.1905
2.3307e-04
13
0.99821
0.99931
0.99905
0.99849
7.5229e-04
7.6118e-04
2.6574e-04
0.0018
-4078.54
1313.75
1.3012e-04
52
104
Appendices
Network
1-14-1
1-15-1
1-20-1
R2tr
R2ts
R2val
R2all
MSEtr
MSEts
MSEval
MSEall
AIC
fitness
MSEModel
Iteration
0.99806
0.99942
0.99901
0.99841
8.1291e-04
3.9929e-04
3.7183e-04
0.0016
-4150.04
2504.445
1.6074e-04
11
0.99918
0.99907
0.99902
0.99914
4.4384e-04
0.0022
4.7351e-04
0.0031
-3714.84
454.5455
2.0593e-04
8
0.99919
0.99911
0.99873
0.99911
4.5875e-04
2.7915e-04
0.0021
0.0028
-3781.81
3582.303
1.9189e-04
21
0.9992
0.99861
0.99904
0.9991
4.5896e-04
3.9718e-04
0.0022
0.0030
-3736.42
2517.75
1.7133e-04
9
0.99824
0.99881
0.99931
0.9985
7.6593e-04
4.1797e-04
4.6437e-04
0.0016
-4150.04
2392.516
1.8553e-04
48
0.99813
0.99913
0.99937
0.99846
7.8217e-04
5.1179e-04
3.4708e-04
0.0016
-4144.04
1953.926
1.4042e-04
24
0.99813
0.99911
0.99931
0.99845
7.8193e-04
3.8085e-04
5.0272e-04
0.0017
-4104.15
2625.706
1.8987e-04
13
0.99909
0.99898
0.99945
0.99914
4.5000e-04
0.0020
3.9654e-04
0.0028
-3775.81
500
2.0830e-04
48
0.99834
0.99945
0.99852
0.99851
7.5409e-04
5.0255e-04
4.3193e-04
0.0017
-4104.15
1989.852
1.9745e-04
57
0.99923
0.99924
0.99909
0.99921
4.1708e-04
4.4432e-04
0.0021
0.0030
-3730.42
2250.63
2.3813e-04
15
0.99814
0.99935
0.99929
0.99846
8.0030e-04
3.0401e-04
4.6247e-04
0.0016
-4114.04
3289.365
1.7275e-04
27
0.99932
0.9991
0.99926
0.99928
3.7190e-04
5.8926e-04
0.0021
0.0031
-3678.84
1697.044
2.6103e-04
40
0.99903
0.999
0.99471
0.9984
8.2699e-04
3.1421e-04
3.2313e-04
0.0015
-4156.51
3182.585
1.6588e-04
41
0.99908
0.99925
0.99943
0.99914
4.3382e-04
0.0021
4.2948e-04
0.0029
-3722.72
476.1905
1.4436e-04
14
0.9992
0.99912
0.99913
0.99918
4.2359e-04
0.0019
5.1976e-04
0.0029
-3722.72
526.3158
2.6546e-04
117
105
Appendices
Network
1-25-1
R2tr
R2ts
R2val
R2all
MSEtr
MSEts
MSEval
MSEall
AIC
fitness
MSEModel
Iteration
0.99912
0.99944
0.99929
0.99919
4.1545e-04
0.0020
5.9629e-04
0.0030
-3670.42
500
1.8548e-04
11
0.99919
0.9934
0.99918
0.99849
7.9213e-04
3.0636e-04
5.2695e-04
0.0016
-4084.04
3264.134
1.5097e-04
19
0.99916
0.99514
0.99939
0.99851
7.4764e-04
5.6775e-04
4.3038e-04
0.0017
-4044.15
1761.339
4.8283e-05
20
0.99821
0.99948
0.99908
0.99852
7.6949e-04
5.1933e-04
4.3467e-04
0.0017
-4044.15
1925.558
1.4090e-04
15
0.99819
0.99906
0.99912
0.99846
7.7864e-04
3.2170e-04
6.0289e-04
0.0017
-4044.15
3108.486
1.5163e-04
28
106
Appendices
Appendices A: Represents finding the best architecture of RNN model by activation functions [TansigPurline] Network
1-5-1
1-6-1
1-7-1
R2tr
R2ts
R2val
R2all
MSEtr
MSEts
MSEval
MSEall
AIC
fitness
MSEModel
Iteration
0.99829
0.99916
0.999
0.99851
7.7739e-04
4.6865e-04
4.2319e-04
0.0017
-4164.15
2133.789
2.1268e-04
12
0.9992
0.99924
0.99365
0.99842
7.7985e-04
3.8038e-04
4.6723e-04
0.0016
-4204.04
2628.95
1.8149e-04
13
0.99915
0.9992
0.99902
0.99914
4.2472e-04
4.9933e-04
0.0020
0.0030
-3790.42
2002.684
1.9119e-04
26
0.99928
0.99934
0.99898
0.99924
3.8846e-04
0.0023
2.8254e-04
0.0030
-3790.42
434.7826
1.5235e-04
148
0.99912
0.99942
0.99951
0.99921
4.1293e-04
4.5755e-04
0.0021
0.0030
-3790.42
2185.553
1.2642e-04
10
0.9982
0.99881
0.99901
0.9984
8.2027e-04
4.7474e-04
3.2835e-04
0.0016
-4198.04
2106.416
1.2762e-04
10
0.99906
0.99928
0.99943
0.99915
4.3199e-04
4.9320e-04
0.0020
0.0029
-3806.72
2027.575
1.5496e-04
34
0.99908
0.9994
0.99524
0.99847
7.6095e-04
5.1297e-04
4.5044e-04
0.0017
-4158.15
1949.432
1.1717e-04
14
0.99924
0.9991
0.99911
0.9992
4.1144e-04
0.0020
5.5679e-04
0.0030
-3784.42
500
1.8002e-04
26
0.99826
0.99919
0.99916
0.99857
7.4077e-04
4.1025e-04
6.6641e-04
0.0018
-4120.54
2437.538
2.2879e-04
21
0.99815
0.99917
0.99892
0.99845
8.0797e-04
3.4813e-04
3.8506e-04
0.0015
-4234.51
2872.49
2.0502e-04
14
0.99921
0.99899
0.999
0.99913
4.4515e-04
4.1270e-04
0.0020
0.0028
-3823.81
2423.068
1.2921e-04
494
0.99897
0.99949
0.99948
0.99912
4.4767e-04
0.0020
4.4738e-04
0.0029
-3800.72
500
1.4438e-04
11
0.99827
0.99897
0.99907
0.9985
7.8792e-04
4.3007e-04
4.6148e-04
0.0017
-4152.15
2325.203
1.9701e-04
9
0.99832
0.99912
0.99923
0.99857
7.2100e-04
4.7496e-04
5.6115e-04
0.0018
-4114.54
2105.44
2.3001e-04
284
107
Appendices
Network
1-8-1
1-9-1
1-10-1
R2tr
R2ts
R2val
R2all
MSEtr
MSEts
MSEval
MSEall
AIC
fitness
MSEModel
Iteration
0.99926
0.99893
0.99895
0.99917
4.2412e-04
2.9983e-04
0.0023
0.0030
-3772.42
3335.223
1.4906e-04
11
0.99914
0.99934
0.99938
0.9992
4.1382e-04
0.0020
5.4949e-04
0.0030
-3772.42
500
1.6522e-04
43
0.99909
0.99418
0.9993
0.99837
8.2043e-04
3.9829e-04
2.9294e-04
0.0015
-4228.51
2510.733
1.3568e-04
12
0.99922
0.99929
0.99878
0.99915
4.2758e-04
3.3901e-04
0.0021
0.0029
-3794.72
2949.765
1.7997e-04
213
0.99915
0.99501
0.99908
0.99844
8.1807e-04
4.4722e-04
2.6525e-04
0.0015
-4228.51
2236.036
1.2252e-04
10
0.99809
0.99942
0.99916
0.99844
8.2019e-04
3.3903e-04
4.0720e-04
0.0016
-4180.04
2949.591
1.7606e-04
13
0.99833
0.99925
0.99897
0.99854
7.6421e-04
3.3719e-04
5.2901e-04
0.0016
-4180.04
2965.687
1.9960e-04
135
0.99823
0.99874
0.99928
0.99824
7.9334e-04
3.7635e-04
4.0778e-04
0.0016
-4180.04
2657.101
1.7546e-04
22
0.99917
0.9993
0.99929
0.99921
4.1204e-04
5.4294e-04
0.0020
0.0030
-3766.42
1841.824
1.6625e-04
26
0.99907
0.9994
0.99935
0.99914
4.3714e-04
0.0021
4.5545e-04
0.0030
-3766.42
476.1905
1.2982e-04
11
0.99817
0.99939
0.99875
0.99281
7.5869e-04
4.9943e-04
4.9689e-04
0.0018
-4096.54
2002.283
1.7472e-04
9
0.99805
0.99943
0.9993
0.99846
7.6244e-04
5.5977e-04
4.5584e-04
0.0018
-4096.54
1786.448
2.0191e-04
25
0.99805
0.9993
0.99935
0.99843
8.2095e-04
3.7033e-04
2.9817e-04
0.0015
-4216.51
2700.294
1.9324e-04
11
0.99808
0.99912
0.99949
0.99844
7.9732e-04
4.4734e-04
3.7036e-04
0.0016
-4174.04
2235.436
1.7378e-04
12
0.9991
0.99881
0.99936
0.99911
4.3867e-04
3.5109e-04
0.0021
0.0029
-3782.72
2848.273
1.8726e-04
15
108
Appendices
Network
1-11-1
1-12-1
1-13-1
R2tr
R2ts
R2val
R2all
MSEtr
MSEts
MSEval
MSEall
AIC
fitness
MSEModel
Iteration
0.99906
0.99906
0.99925
0.99908
5.2744e-04
0.0021
5.6921e-04
0.0032
-3711.95
476.1905
3.5272e-04
9
0.99914
0.99928
0.99861
0.9991
4.6819e-04
3.3805e-04
0.0020
0.0028
-3799.81
2958.142
1.5042e-04
39
0.99807
0.99927
0.99888
0.99839
8.0844e-04
4.8243e-04
2.9982e-04
0.0016
-4168.04
2072.84
1.6449e-04
10
0.99911
0.99938
0.99463
0.99847
7.6720e-04
5.3966e-04
3.8617e-04
0.0017
-4128.15
1853.019
1.8630e-04
13
0.99822
0.999
0.99949
0.99856
7.5224e-04
4.3033e-04
4.9506e-04
0.0017
-4128.15
2323.798
2.0921e-04
192
0.99918
0.99955
0.99512
0.99859
7.3794e-04
5.0009e-04
6.3089e-04
0.0019
-4048.96
1999.64
1.2759e-04
9
0.99818
0.99919
0.99896
0.99842
7.8791e-04
3.5849e-04
5.1986e-04
0.0017
-4122.15
2789.478
2.2403e-04
11
0.99924
0.99894
0.99874
0.99913
4.2820e-04
4.4153e-04
0.0021
0.0029
-3770.72
2264.852
1.6161e-04
10
0.99925
0.99904
0.99914
0.99921
4.1589e-04
0.0022
3.9294e-04
0.0030
-3748.42
454.5455
1.6484e-04
24
0.99926
0.9943
0.99897
0.99849
7.8839e-04
3.2301e-04
5.1418e-04
0.0016
-4162.04
3095.879
1.5229e-04
12
0.99829
0.99905
0.99911
0.99851
7.5746e-04
5.0454e-04
4.8989e-04
0.0018
-4078.54
1982.003
1.9000e-04
9
0.99809
0.99943
0.99937
0.99849
7.9077e-04
4.4702e-04
4.2589e-04
0.0017
-4116.15
2237.036
1.7651e-04
10
0.99926
0.99915
0.99892
0.99919
4.5141e-04
4.6455e-04
0.0022
0.0031
-3720.84
2152.621
5.9640e-05
8
0.9991
0.99504
0.99936
0.9985
7.7441e-04
5.2406e-04
3.2311e-04
0.0016
-4156.04
1908.178
1.7859e-04
60
0.99919
0.99919
0.99344
0.99839
8.0789e-04
3.4745e-04
4.0632e-04
0.0016
-4156.04
2878.112
1.7372e-04
15
109
Appendices
Network
1-14-1
1-15-1
1-20-1
R2tr
R2ts
R2val
R2all
MSEtr
MSEts
MSEval
MSEall
AIC
fitness
MSEModel
Iteration
0.99827
0.99928
0.99938
0.99857
7.4057e-04
6.0531e-04
5.0714e-04
0.0019
-4036.96
1652.046
1.9645e-04
15
0.99804
0.99933
0.99914
0.9984
7.6354e-04
3.3113e-04
5.3941e-04
0.0016
-4150.04
3019.962
1.4360e-04
63
0.99923
0.9986
0.99907
0.99911
4.4555e-04
3.5093e-04
0.0021
0.0029
-3758.72
2849.571
1.4926e-04
20
0.99923
0.99939
0.99451
0.99854
7.6665e-04
4.5007e-04
5.9287e-04
0.0018
-4072.54
2221.877
2.4078e-04
8
0.99804
0.99927
0.99938
0.99843
8.0129e-04
4.2287e-04
3.9544e-04
0.0016
-4150.04
2364.793
1.6529e-04
9
0.99919
0.99923
0.9939
0.9985
7.7500e-04
4.3718e-04
4.5431e-04
0.0017
-4104.15
2287.387
1.7397e-04
12
0.99906
0.99911
0.99946
0.99913
4.3537e-04
4.5748e-04
0.0021
0.0030
-3730.42
2185.888
1.4468e-04
13
0.99807
0.99924
0.99963
0.99848
8.0563e-04
4.1360e-04
3.6501e-04
0.0016
-4144.04
2417.795
1.8222e-04
9
0.99913
0.99877
0.99941
0.99941
0.0005
4.9501e-04
0.0019
0.0029
-3752.72
2020.161
1.2503e-04
14
0.99918
0.99893
0.9994
0.99916
4.3222e-04
4.0297e-04
0.0020
0.0029
-3752.72
2481.574
1.4574e-04
99
0.99817
0.99909
0.99905
0.99845
7.7467e-04
2.7052e-04
5.9966e-04
0.0016
-4114.04
3696.584
1.5277e-04
29
0.99897
0.9994
0.99919
0.99907
4.7735e-04
0.0020
3.0650e-04
0.0027
-3769.74
500
1.6527e-04
30
0.99808
0.99915
0.99932
0.99843
7.8693e-04
3.7840e-04
4.4425e-04
0.0016
-4114.04
2642.706
1.9144e-04
23
0.99839
0.99882
0.99913
0.99857
7.4294e-04
5.0980e-04
5.7260e-04
0.0018
-4036.54
1961.554
2.4132e-04
39
0.99815
0.9995
0.99899
0.99847
7.8392e-04
4.4067e-04
4.1118e-04
0.0016
-4114.04
2269.272
1.7233e-04
15
110
Appendices
Network
1-25-1
R2tr
R2ts
R2val
R2all
MSEtr
MSEts
MSEval
MSEall
AIC
fitness
MSEModel
Iteration
0.99912
0.99501
0.99913
0.99844
7.6644e-04
3.7187e-04
6.2930e-04
0.0018
-4006.54
2689.112
1.1743e-04
8
0.99809
0.99882
0.99908
0.99834
8.2055e-04
3.1999e-04
3.9561e-04
0.0015
-4126.51
3125.098
1.7850e-04
14
0.99808
0.9988
0.99964
0.99842
7.8672e-04
4.8573e-04
4.0583e-04
0.0017
-4044.15
2058.757
2.6649e-04
76
0.99913
0.99942
0.99462
0.99853
7.6377e-04
3.5042e-04
5.1462e-04
0.0016
-4084.04
2853.718
1.8857e-04
67
0.99907
0.9994
0.99919
0.99914
4.2681e-04
5.0025e-04
0.0020
0.0029
-3692.72
1999
1.4063e-04
31
111
Appendices
Appendices A: Represents finding the best architecture of RNN model by activation functions [LogsigTansigPurline] Network
R2tr
R2ts
R2val
R2all
MSEtr
MSEts
MSEval
MSEall
AIC
fitness
MSEModel
Iteration
1-5-5-1
0.99925
0.99885
0.99942
0.99921
4.0208e-04
4.2414e-04
0.0022
0.0030
-3742.42
2357.712
1.4402e-04
29
0.99905
0.99944
0.99561
0.9985
7.7578e-04
5.0690e-04
4.1774e-04
0.0017
-4116.15
1972.776
1.4263e-04
126
0.99923
0.99927
0.99882
0.99915
4.2706e-04
5.6545e-04
0.0020
0.0030
-3742.42
1768.503
1.1440e-04
12
0.99919
0.99883
0.99896
0.99911
4.3713e-04
0.0021
3.9831e-04
0.0029
-3694.72
476.1905
1.5077e-04
42
0.99901
0.99938
0.99897
0.99906
5.1708e-04
3.2968e-04
0.0021
0.0030
-3672.42
3033.244
1.6833e-04
18
0.99797
0.99925
0.99939
0.99834
8.1896e-04
3.4009e-04
3.4344e-04
0.0015
-4128.51
2940.398
1.6300e-04
15
0.99819
0.99889
0.99897
0.99842
8.0265e-04
3.2252e-04
4.6827e-04
0.0016
-4096.04
3100.583
1.8017e-04
70
0.99822
0.99918
0.9989
0.99846
7.7000e-04
4.5068e-04
4.6454e-04
0.0017
-4056.15
2218.869
1.8085e-04
11
0.99803
0.99921
0.99946
0.99841
7.9699e-04
3.1618e-04
4.4365e-04
0.0016
-4096.04
3162.755
1.8798e-04
60
0.99827
0.99883
0.99905
0.99847
7.7561e-04
5.2319e-04
3.7639e-04
0.0017
-3936.15
1911.352
1.8457e-04
64
0.999
0.99945
0.99931
0.99911
4.5063e-04
2.7795e-04
0.0021
0.0029
-3584.72
3597.769
1.3201e-04
37
0.99928
0.999
0.9988
0.99916
4.3187e-04
4.6693e-04
0.0021
0.0030
-3562.42
2141.649
2.0851e-04
11
0.9991
0.99944
0.99929
0.99916
4.2703e-04
4.1534e-04
0.0021
0.0029
-3494.72
2407.666
1.7329e-04
75
0.99923
0.99895
0.99442
0.9984
8.1319e-04
3.3267e-04
4.9103e-04
0.0016
-3886.04
3005.982
1.4591e-04
16
0.99911
0.99935
0.99913
0.99913
4.4589e-04
4.1711e-04
0.0020
0.0029
-3494.72
2397.449
1.3597e-04
77
1-10-5-1
1-5-10-1
1-10-10-1
1-15-10-1
112
Appendices
Network
1-10-15-1
1-15-15-1
R2tr
R2ts
R2val
R2all
MSEtr
MSEts
MSEval
MSEall
AIC
fitness
MSEModel
Iteration
0.99914
0.99904
0.99508
0.99846
7.7199e-04
4.9307e-04
4.0822e-04
0.0017
-3846.15
2028.11
1.6410e-04
12
0.9991
0.99893
0.99374
0.99834
8.2518e-04
4.6936e-04
6.1930e-04
0.0019
-3772.96
2130.561
1.4872e-04
13
0.99819
0.99917
0.99864
0.9984
7.9377e-04
4.0463e-04
4.7421e-04
0.0017
-3846.15
2471.394
1.5846e-04
21
0.9982
0.99927
0.9991
0.99851
7.6257e-04
3.7753e-04
5.6297e-04
0.0017
-3686.15
2648.796
1.7949e-04
22
0.99823
0.99918
0.99946
0.99855
7.6044e-04
4.9042e-04
4.5026e-04
0.0017
-3686.15
2039.069
1.8183e-04
59
0.99921
0.99424
0.99
0.99
7.8228e-04
4.7707e-04
3.8975e-04
0.0016
-3726.04
2096.128
1.4790e-04
14
113
Appendices
Appendices A: Represents finding the best architecture of RNN model by activation functions [LogsigLogsigPurline] Network
1-5-5-1
1-10-5-1
1-5-10-1
1-10-10-1
1-15-10-1
R2tr
R2ts
R2val
R2all
MSEtr
MSEts
0.99915
0.99915
0.99916
0.99915
4.2710e-04
4.9378e-04
0.0021
0.99839
0.99904
0.99895
0.99855
7.5481e-04
5.0683e-04
0.9992
0.99918
0.99463
0.99848
7.8198e-04
0.99826
0.99889
0.99924
0.99849
0.999
0.9945
0.99951
0.99918
0.99885
0.99816
MSEval MSEall
AIC
fitness
MSEModel Iteration
0.0030
-3742.42
2025.193
1.5656e-04
30
4.0040e-04
0.0017
-4116.15
1973.048
1.7675e-04
38
3.8307e-04
4.7856e-04
0.0016
-4156.04
2610.489
1.8800e-04
197
7.7023e-04
5.0596e-04
4.0973e-04
0.0017
-4046.15
1976.441
2.0935e-04
19
0.99841
7.9415e-04
4.1903e-04
3.5593e-04
0.0016
-4086.04
2386.464
1.5680e-04
23
0.99925
0.99915
4.1578e-04
3.4932e-04
0.0022
0.0030
-3672.42
2862.705
1.7391e-04
28
0.9992
0.99886
0.99843
7.9476e-04
2.9259e-04
5.4368e-04
0.0016
-4096.04
3417.752
2.1784e-04
16
0.99794
0.99931
0.99917
0.99836
8.3111e-04
2.7441e-04
3.5290e-04
0.0015
-4138.51
3644.182
1.5748e-04
23
0.99923
0.99922
0.99403
0.99847
7.5477e-04
6.8718e-04
3.2002e-04
0.0018
-4018.54
1455.223
1.8670e-04
14
0.9992
0.99341
0.99926
0.99843
8.0921e-04
3.8072e-04
3.6067e-04
0.0016
-3976.04
2626.602
1.8019e-04
37
0.9992
0.99933
0.9945
0.99855
7.5468e-04
5.2554e-04
5.5394e-04
0.0018
-3898.54
1902.805
1.4675e-04
14
0.99915
0.99879
0.99919
0.9991
4.5363e-04
2.8104e-04
0.0022
0.0029
-3584.72
3558.212
1.1025e-04
9
0.99915
0.99871
0.99929
0.99911
4.5403e-04
0.0021
3.5654e-04
0.0029
-3494.72
476.1905
1.1840e-04
15
0.99919
0.99912
0.99486
0.99849
7.9238e-04
2.9781e-04
5.7360e-04
0.0017
-3846.15
3357.846
1.6157e-04
36
0.99827
0.99889
0.99932
0.99853
7.4127e-04
6.2592e-04
3.9484e-04
0.0018
-3808.54
1597.648
1.7927e-04
17
114
Appendices
Network
1-10-15-1
1-15-15-1
R2tr
R2ts
R2val
R2all
MSEtr
MSEts
MSEval
MSEall
AIC
fitness
MSEModel
Iteration
0.99823
0.99923
0.99922
0.99854
7.5554e-04
3.8988e-04
5.3309e-04
0.0017
-3846.15
2564.892
1.9447e-04
91
0.99916
0.9942
0.99937
0.99853
7.5095e-04
5.0804e-04
4.4638e-04
0.0017
-3846.15
1968.349
1.7540e-04
147
0.99819
0.99939
0.99947
0.99857
7.3838e-04
5.8481e-04
4.8945e-04
0.0018
-3808.54
1709.957
2.1027e-04
12
0.99905
0.99945
0.99932
0.99915
4.5717e-04
0.0019
5.9334e-04
0.0030
-3312.42
526.3158
2.8050e-04
13
0.99919
0.99894
0.99939
0.99918
4.2260e-04
5.1199e-04
0.0021
0.0030
-3312.42
1953.163
1.2520e-04
15
0.99925
0.99908
0.99452
0.99852
7.6849e-04
4.4807e-04
3.8622e-04
0.0016
-3726.04
2231.794
1.8652e-04
140
115
Appendices
Appendices A: Represents finding the best architecture of RNN model by activation functions [TansigLogsigPurline] Network
1-5-5-1
1-10-5-1
1-5-10-1
1-10-10-1
1-15-10-1
R2tr
R2ts
R2val
R2all
MSEtr
MSEts
MSEval
MSEall
AIC
fitness
MSEModel
Iteration
0.99816
0.99932
0.99902
0.99845
8.0057e-04
3.6163e-04
4.2288e-04
0.0016
-4156.04
2765.257
2.2994e-04
13
0.99821
0.99916
0.99843
0.99944
7.8416e-04
4.3690e-04
3.3885e-04
0.0016
-4156.04
2288.853
1.5247e-04
27
0.99929
0.99456
0.99924
0.99857
7.4100e-04
4.2141e-04
5.6346e-04
0.0017
-4116.15
2372.986
1.8681e-04
69
0.99921
0.9991
0.99891
0.99914
4.3873e-04
4.8462e-04
0.0019
0.0029
-3694.72
2063.472
1.3972e-04
81
0.99824
0.99925
0.99908
0.99852
7.4791e-04
5.9869e-04
4.0971e-04
0.0018
-4008.54
1670.314
1.4711e-04
16
0.99906
0.99928
0.99948
0.99916
4.3290e-04
4.8883e-04
0.0020
0.0029
-3694.72
2045.701
1.7206e-04
15
0.99834
0.9992
0.99882
0.99853
7.6608e-04
4.6284e-04
5.0773e-04
0.0017
-4056.15
2160.574
2.3377e-04
11
0.9992
0.99936
0.99946
0.99926
3.8744e-04
5.5592e-04
0.0021
0.0031
-3660.84
1798.82
1.7144e-04
13
0.99824
0.99942
0.99929
0.99862
7.1201e-04
5.4034e-04
6.7441e-04
0.0019
-3982.96
1850.687
1.9159e-04
14
0.99912
0.99941
0.99934
0.9992
4.0049e-04
0.0022
4.4900e-04
0.0030
-3562.42
454.5455
1.4535e-04
16
0.99798
0.99954
0.99922
0.99843
8.0002e-04
2.8818e-04
4.8037e-04
0.0016
-3976.04
3470.053
1.5467e-04
11
0.99921
0.99885
0.99866
0.99907
4.6841e-04
2.7665e-04
0.0020
0.0027
-3631.74
3614.676
1.7478e-04
12
0.99923
0.99909
0.99928
0.9992
4.1227e-04
4.2571e-04
0.0021
0.0030
-3472.42
2349.017
1.6175e-04
32
0.99812
0.99921
0.99925
0.99842
7.8225e-04
3.2785e-04
4.9582e-04
0.0016
-3886.04
3050.175
1.9211e-04
17
0.99825
0.99899
0.99927
0.9985
7.5816e-04
6.2673e-04
3.6076e-04
0.0017
-3846.15
1595.583
1.8911e-04
15
116
Appendices
Network
1-10-15-1
1-15-15-1
R2tr
R2ts
R2val
R2all
MSEtr
MSEts
MSEval
MSEall
AIC
fitness
MSEModel
Iteration
0.99902
0.99908
0.99932
0.99908
4.7447e-04
3.5523e-04
0.0020
0.0028
-3517.81
2815.078
2.4196e-04
10
0.99829
0.99915
0.99922
0.99856
7.2559e-04
5.8160e-04
4.9161e-04
0.0018
-3808.54
1719.395
2.1660e-04
46
0.99902
0.99953
0.99941
0.99917
4.1603e-04
4.4449e-04
0.0020
0.0029
-3494.72
2249.769
1.4601e-04
38
0.99912
0.99549
0.99939
0.99852
7.7698e-04
5.6947e-04
3.1411e-04
0.0017
-3686.15
1756.019
1.1635e-04
12
0.99913
0.99934
0.99926
0.99918
4.2037e-04
0.0021
3.8141e-04
0.0029
-3334.72
476.1905
2.1071e-04
19
0.99914
0.99929
0.99919
0.99916
4.1750e-04
4.4667e-04
0.0020
0.0029
-3334.72
2238.789
1.6426e-04
12
117
Appendices
Appendices A: Represents finding the best architecture of RNN model by activation functions [TansigTansigPurline] Network
1-5-5-1
1-10-5-1
1-5-10-1
1-10-10-1
1-15-10-1
R2tr
R2ts
R2val
R2all
MSEtr
MSEts
MSEval
MSEall
AIC
fitness
MSEModel
Iteration
0.99814
0.99934
0.999
0.99843
8.0889e-04
3.5153e-04
4.2158e-04
0.0016
-4156.04
2844.707
2.1825e-04
17
0.99831
0.9992
0.99885
0.99851
7.6703e-04
5.4131e-04
3.6627e-04
0.0017
-4116.15
1847.37
1.7878e-04
10
0.99819
0.99919
0.99931
0.99852
7.4593e-04
4.9045e-04
4.9120e-04
0.0017
-4116.15
2038.944
1.7714e-04
40
0.99903
0.99941
0.99937
0.99913
4.4896e-04
0.0020
4.2039e-04
0.0028
-3717.81
500
1.7243e-04
18
0.99916
0.99537
0.99844
0.99843
7.7907e-04
4.3493e-04
4.3160e-04
0.0016
-4086.04
2299.221
1.5267e-04
14
0.99824
0.9994
0.99893
0.99852
7.3923e-04
6.0613e-04
4.6902e-04
0.0018
-4008.54
1649.811
1.6019e-04
25
0.99905
0.99902
0.99946
0.9991
0.000478
0.000239
0.00078
0.0015
-4234.51
4176.063
1.2371e-04
10
0.99926
0.99445
0.99911
0.99854
7.5435e-04
4.2836e-04
5.9358e-04
0.0018
-4018.54
2334.485
1.3356e-04
11
0.99922
0.99897
0.99378
0.99848
7.6942e-04
3.9398e-04
4.6546e-04
0.0016
-4096.04
2538.2
1.8253e-04
166
0.99807
0.99911
0.99925
0.9984
7.9607e-04
4.1845e-04
3.8078e-04
0.0016
-3976.04
2389.772
1.6267e-04
11
0.99823
0.99931
0.99854
0.99845
7.7170e-04
3.7777e-04
5.5027e-04
0.0017
-3936.15
2647.113
1.9314e-04
13
0.99918
0.99438
0.99937
0.99848
7.5359e-04
5.0757e-04
4.9111e-04
0.0018
-3898.54
1970.172
1.3616e-04
101
0.99821
0.99948
0.99921
0.99852
7.6425e-04
5.8678e-04
3.4114e-04
0.0017
-3846.15
1704.216
1.7659e-04
48
0.99814
0.99928
0.99925
0.99851
7.3250e-04
6.7132e-04
4.3281e-04
0.0018
-3808.54
1489.603
2.1573e-04
13
0.99807
0.99957
0.9993
0.99849
7.6195e-04
4.3277e-04
5.0312e-04
0.0017
-3846.15
2310.696
2.0717e-04
14
118
Appendices
Network
1-10-15-1
1-15-15-1
R2tr
R2ts
R2val
R2all
MSEtr
MSEts
MSEval
MSEall
AIC
fitness
MSEModel
Iteration
0.99916
0.99921
0.99949
0.99922
3.9868e-04
4.0568e-04
0.0022
0.0030
-3472.42
2464.997
1.6044e-04
14
0.99815
0.99929
0.99928
0.9985
7.9081e-04
4.3623e-04
4.1600e-04
0.0016
-3886.04
2292.369
1.9480e-04
15
0.99808
0.99878
0.99896
0.99837
8.2082e-04
2.6461e-04
4.1121e-04
0.0015
-3928.51
3779.147
2.2669e-04
12
0.99901
0.99932
0.99952
0.99915
4.2691e-04
0.0020
4.7933e-04
0.0029
-3334.72
500
1.5387e-04
14
0.99933
0.99502
0.99891
0.99859
7.4571e-04
6.2139e-04
4.2351e-04
0.0018
-3648.54
1609.295
1.4413e-04
33
0.99816
0.99941
0.99919
0.99852
7.6989e-04
4.8520e-04
4.3461e-04
0.0017
-3686.15
2061.006
1.8167e-04
15
119
Appendices
Appendices B: The average of daily power of load and demand of Sulaymaniyah/ directorate of control & communication in the January of 2013 until July of 2015 No. Load
Demand No.
Load
Demand No.
Load
Demand No.
Load
Demand No. Load
Demand
1
136.54
136.31
16
158.84
157.14
31
145.53
144.01
46
123.32
127.82
61
117.11
115.95
2
139.16
137.83
17
159.36
157.35
32
149.80
149.09
47
122.43
126.94
62
118.01
116.95
3
137.75
136.29
18
151.08
148.41
33
154.67
152.88
48
127.32
126.83
63
115.18
113.84
4
132.93
131.95
19
149.06
147.08
34
152.93
149.92
49
130.30
129.89
64
114.83
113.75
5
136.17
135.84
20
148.46
146.58
35
149.35
146.48
50
131.00
130.25
65
123.26
122.12
6
143.60
143.00
21
146.06
143.74
36
144.32
142.68
51
130.98
129.91
66
134.90
135.13
7
150.30
148.26
22
142.15
139.98
37
143.85
142.44
52
133.82
133.29
67
133.54
132.21
8
153.45
151.08
23
138.48
137.01
38
145.54
144.05
53
126.45
125.33
68
135.32
133.64
9
156.82
154.69
24
133.18
131.70
39
136.92
134.91
54
129.85
128.87
69
125.95
123.31
10
160.71
158.55
25
130.26
129.51
40
133.34
132.22
55
127.65
126.42
70
114.90
113.21
11
160.41
158.33
26
135.35
133.99
41
125.88
124.79
56
123.73
124.47
71
115.04
114.22
12
161.41
159.20
27
136.52
134.75
42
119.39
118.98
57
119.22
118.59
72
107.69
106.69
13
162.27
159.99
28
138.87
137.17
43
125.26
125.45
58
114.48
114.02
73
97.45
97.08
14
159.47
157.24
29
143.20
142.05
44
128.86
127.59
59
113.69
113.39
74
85.73
85.70
15
159.14
157.13
30
143.75
141.50
45
129.05
133.19
60
115.01
114.95
75
85.27
85.29
120
Appendices
Appendices B: The average of daily power of load and demand of Sulaymaniyah/ directorate of control & communication in the January of 2013 until July of 2015 No. Load
Demand No.
Load
Demand No.
Load
Demand No.
Load
Demand No. Load
Demand
76
89.38
89.27
91
85.33
85.32
106
77.83
77.77
121
78.11
78.02
136
76.11
76.22
77
104.88
104.94
92
82.29
82.13
107
78.46
78.45
122
78.76
78.70
137
72.67
72.52
78
109.11
109.42
93
82.75
82.55
108
78.35
78.32
123
72.94
73.14
138
74.62
74.54
79
107.08
107.04
94
82.85
82.85
109
72.89
72.85
124
76.92
77.22
139
76.18
75.99
80
89.17
89.14
95
87.99
88.02
110
78.49
78.58
125
77.80
78.17
140
76.92
77.01
81
92.20
92.21
96
89.51
89.43
111
87.17
86.94
126
77.92
78.24
141
76.14
76.12
82
100.10
101.04
97
82.26
82.02
112
97.65
97.60
127
78.21
78.35
142
77.14
77.03
83
107.54
108.67
98
77.82
77.69
113
97.83
97.50
128
80.27
80.52
143
78.09
78.02
84
109.59
109.84
99
77.20
77.13
114
93.39
93.26
129
79.49
79.22
144
74.15
74.11
85
106.77
106.58
100
78.30
78.59
115
89.19
89.06
130
72.79
72.54
145
83.16
83.24
86
105.47
105.82
101
81.31
81.30
116
78.88
78.81
131
76.68
76.53
146
90.27
90.19
87
101.59
101.41
102
74.50
74.87
117
80.35
80.33
132
78.07
77.86
147
90.25
90.18
88
95.84
95.74
103
80.19
80.19
118
79.30
79.49
133
77.69
77.58
148
88.52
88.41
89
99.43
99.44
104
78.40
78.27
119
78.83
78.97
134
76.27
76.07
149
86.70
86.68
90
91.58
91.49
105
78.24
78.16
120
77.78
77.71
135
76.18
76.11
150
86.48
86.82
121
Appendices
Appendices B: The average of daily power of load and demand of Sulaymaniyah/ directorate of control & communication in the January of 2013 until July of 2015 No. Load
Demand No.
Load
Demand No.
Load
Demand No.
Load
Demand No. Load
Demand
151
82.34
82.62
166
108.01
108.06
181
115.80
115.73
196
121.27
120.78
211 109.49
109.44
152
91.12
91.13
167
107.42
107.32
182
110.87
110.75
197
120.71
120.56
212 109.10
109.10
153
94.86
94.85
168
105.55
105.37
183
110.13
109.83
198
116.40
116.23
213 104.54
104.45
154
96.13
96.11
169
106.47
106.40
184
110.37
110.40
199
106.42
106.38
214 113.87
113.89
155
99.39
99.40
170
105.29
105.20
185
104.53
104.40
200
110.52
110.67
215 118.56
118.23
156
98.22
98.14
171
98.58
98.50
186
114.00
114.03
201
115.94
115.96
216 113.40
113.34
157
95.40
95.31
172
103.61
103.62
187
115.19
114.96
202
119.47
119.36
217 112.33
111.90
158
86.94
86.84
173
107.40
107.14
188
114.54
114.24
203
119.03
118.67
218 116.67
116.74
159
91.54
91.56
174
109.09
108.59
189
120.51
120.33
204
115.87
115.76
219 106.05
106.05
160
93.92
93.79
175
108.84
108.73
190
124.31
123.78
205
113.41
113.23
220 100.34
100.34
161
98.13
98.11
176
110.20
110.05
191
124.38
123.84
206
108.37
108.11
221 101.15
101.16
162
96.92
96.85
177
112.90
112.70
192
112.37
112.25
207
114.85
115.33
222 106.41
106.40
163
98.47
98.55
178
102.33
101.57
193
114.54
114.56
208
109.75
109.54
223 113.23
113.15
164 102.36
102.38
179
106.08
106.14
194
118.68
118.52
209
107.15
107.17
224 117.34
117.14
165 100.30
100.16
180
111.99
111.91
195
125.68
125.54
210
107.77
107.75
225 115.51
115.34
122
Appendices
Appendices B: The average of daily power of load and demand of Sulaymaniyah/ directorate of control & communication in the January of 2013 until July of 2015 No. Load
Demand No.
Load
Demand No. Load
Demand
No. Load
Demand No. Load
Demand
226 117.55
117.32
241 102.46
102.44
256 92.55
92.56
271 78.50
78.47
286 78.65
78.67
227 110.67
110.64
242 106.55
106.50
257 93.18
93.17
272 78.42
78.39
287 65.01
65.00
228 116.95
116.91
243 108.05
108.97
258 95.11
95.07
273 78.26
78.24
288 64.50
64.51
229 118.64
118.03
244 111.04
110.70
259 92.23
92.22
274 78.36
78.17
289 65.29
65.27
230 114.26
112.28
245 113.20
112.79
260 90.59
90.56
275 79.09
79.07
290 66.78
66.78
231 112.78
112.67
246 111.21
111.15
261 89.16
89.13
276 75.99
75.68
291 71.63
71.66
232 111.63
111.52
247 108.56
108.57
262 82.78
82.80
277 75.90
75.90
292 72.24
72.41
233 110.86
110.74
248 95.42
95.36
263 79.77
79.77
278 73.70
73.63
293 75.50
75.68
234 104.68
104.70
249 92.78
92.89
264 87.46
87.45
279 73.09
72.95
294 76.01
76.00
235 112.39
112.41
250 93.87
93.67
265 86.08
85.96
280 73.57
73.62
295 75.26
74.87
236 109.13
109.05
251 94.59
94.57
266 83.66
83.57
281 75.03
75.31
296 75.82
75.76
237 105.78
105.57
252 95.14
95.02
267 78.45
78.41
282 73.04
72.90
297 74.66
74.59
238 106.45
106.34
253 95.18
95.10
268 76.80
76.78
283 71.20
71.25
298 74.91
74.86
239 109.29
109.26
254 94.49
94.48
269 74.74
74.73
284 70.21
70.16
299 75.74
75.70
240 111.50
110.88
255 87.65
87.47
270 77.35
77.44
285 71.69
71.68
300 79.21
79.14
123
Appendices
Appendices B: The average of daily power of load and demand of Sulaymaniyah/ directorate of control & communication in the January of 2013 until July of 2015 No. Load
Demand No.
Load
Demand No.
Load
Demand No.
Load
Demand No. Load
Demand
301
80.97
80.49
316
97.22
96.96
331
109.71
109.35
346
150.58
147.98
361
148.97
146.81
302
79.98
79.83
317
96.60
96.50
332
111.61
111.52
347
155.40
152.86
362
148.25
145.39
303
79.57
79.53
318
96.37
96.11
333
110.27
109.98
348
155.20
151.78
363
147.59
146.70
304
78.47
78.38
319
103.63
103.54
334
114.01
113.04
349
154.45
152.06
364
148.20
146.34
305
82.70
82.60
320
105.24
104.87
335
115.61
115.05
350
152.90
150.71
365
146.94
145.13
306
85.43
85.34
321
106.44
106.09
336
115.97
115.03
351
155.09
152.59
366
147.15
143.83
307
84.80
84.61
322
106.64
106.60
337
122.70
122.01
352
158.04
154.92
367
145.49
142.97
308
87.14
87.15
323
113.31
113.28
338
132.08
131.44
353
153.63
151.84
368
149.01
145.86
309
87.95
87.94
324
114.50
114.17
339
134.13
132.30
354
156.06
153.82
369
154.40
153.24
310
87.37
87.22
325
108.74
108.47
340
134.80
132.26
355
156.37
153.19
370
157.16
154.95
311
94.07
93.90
326
107.33
107.11
341
139.62
138.15
356
154.69
153.13
371
157.36
153.69
312
97.77
98.45
327
107.10
106.65
342
140.40
137.93
357
151.30
148.74
372
154.88
150.65
313 106.96
106.71
328
108.32
107.81
343
147.39
145.90
358
148.57
146.30
373
151.59
148.00
314 104.98
104.70
329
108.99
108.78
344
151.70
150.03
359
145.82
144.14
374
149.85
147.82
315
98.47
330
107.60
107.26
345
153.66
152.00
360
146.75
145.44
375
151.88
149.84
98.88
124
Appendices
Appendices B: The average of daily power of load and demand of Sulaymaniyah/ directorate of control & communication in the January of 2013 until July of 2015 No. Load
Demand No.
Load
Demand No.
Load
Demand No.
Load
Demand No. Load
Demand
376 149.88
145.56
391
137.78
135.95
406
146.35
143.38
421
109.61
108.49
436
124.97
125.41
377 150.66
149.49
392
141.60
139.10
407
143.69
140.21
422
109.06
107.02
437
126.80
123.38
378 151.01
146.94
393
139.15
135.31
408
137.89
137.89
423
112.47
111.84
438
122.74
119.18
379 149.03
146.44
394
135.98
132.59
409
130.43
127.02
424
119.16
117.22
439
118.97
116.41
380 143.99
140.00
395
130.33
127.84
410
130.20
127.53
425
118.00
115.09
440
121.75
120.73
381 135.86
132.82
396
140.70
140.57
411
132.31
130.54
426
114.27
112.42
441
121.79
118.90
382 136.96
134.65
397
151.04
150.22
412
133.28
130.46
427
117.23
115.74
442
114.19
111.72
383 135.61
133.37
398
158.16
158.36
413
131.63
128.64
428
113.93
110.58
443
105.02
104.79
384 133.85
132.11
399
155.03
151.41
414
128.84
125.55
429
106.80
104.41
444
82.99
82.96
385 134.35
132.33
400
157.81
153.61
415
125.71
122.65
430
94.17
93.35
445
95.39
95.46
386 132.48
130.92
401
161.14
158.90
416
118.50
115.63
431
91.13
90.89
446
97.77
97.74
387 130.73
130.08
402
154.20
151.12
417
119.61
117.62
432
94.04
94.05
447
94.33
94.12
388 129.53
127.56
403
152.42
148.98
418
115.33
112.78
433
101.94
101.17
448
91.73
91.30
389 133.31
130.83
404
149.49
146.73
419
112.96
112.12
434
101.80
101.62
449
91.03
90.61
390 134.36
132.09
405
149.19
145.91
420
108.10
106.19
435
107.73
106.78
450
90.21
89.62
125
Appendices
Appendices B: The average of daily power of load and demand of Sulaymaniyah/ directorate of control & communication in the January of 2013 until July of 2015 No. Load
Demand No.
Load
Demand No.
Load
Demand No.
Load
Demand No. Load
Demand
451
81.71
81.63
466
85.91
85.92
481
73.39
73.34
496
80.15
80.20
511
79.32
79.12
452
87.38
87.32
467
88.41
88.33
482
73.99
73.94
497
76.49
76.64
512
80.17
80.10
453
95.66
100.65
468
94.31
94.06
483
74.10
74.31
498
74.85
74.79
513
82.73
82.61
454 115.13
115.62
469
90.34
90.30
484
67.62
67.61
499
74.59
74.46
514
81.10
80.93
455 114.69
113.38
470
84.55
84.54
485
69.72
69.72
500
72.44
72.42
515
89.95
90.42
456 114.03
112.72
471
81.24
81.10
486
66.96
66.94
501
80.55
80.58
516
93.77
93.86
457 106.92
106.19
472
71.96
71.80
487
72.64
72.64
502
83.31
83.10
517
89.05
88.01
458
97.36
97.11
473
74.24
74.26
488
72.08
72.04
503
79.46
79.38
518
84.85
84.72
459 100.75
100.71
474
73.27
73.23
489
73.96
73.95
504
77.91
77.85
519
83.72
83.57
460
93.65
94.01
475
73.44
73.42
490
76.90
76.90
505
76.08
76.06
520
84.21
84.17
461
88.52
87.76
476
73.94
73.92
491
80.20
79.90
506
74.88
74.81
521
81.31
81.28
462
89.71
89.32
477
73.69
73.48
492
81.98
81.63
507
69.53
69.51
522
87.14
87.15
463
85.83
85.74
478
74.12
73.98
493
78.14
77.96
508
76.70
76.76
523
92.09
91.74
464
95.65
95.70
479
69.13
69.10
494
82.35
82.34
509
81.50
81.21
524
96.33
95.85
465
87.07
87.01
480
74.48
74.50
495
81.15
81.06
510
81.61
81.14
525
100.28
99.70
126
Appendices
Appendices B: The average of daily power of load and demand of Sulaymaniyah/ directorate of control & communication in the January of 2013 until July of 2015 No. Load
Demand No.
Load
Demand No.
Load
Demand No.
Load
Demand No. Load
Demand
526 104.18
103.75
541
104.31
103.49
556
104.13
103.48
571
119.65
119.03
586
113.73
113.35
527
95.84
95.36
542
101.78
101.04
557
113.30
113.97
572
115.31
114.04
587
113.94
113.02
528
84.94
85.20
543
112.16
112.65
558
119.01
118.07
573
103.46
103.43
588
113.05
111.87
529
89.14
89.50
544
117.32
116.02
559
116.12
114.97
574
99.39
99.38
589
113.86
113.02
530
94.34
94.58
545
115.40
114.13
560
117.09
115.88
575
101.61
101.63
590
117.94
118.07
531
97.29
97.49
546
115.72
113.80
561
118.70
117.44
576
106.99
106.86
591
115.64
114.43
532 100.80
100.57
547
120.32
120.36
562
122.83
122.12
577
107.95
107.96
592
121.37
121.70
533 104.36
103.85
548
123.15
122.14
563
117.75
115.99
578
116.24
116.39
593
123.22
121.85
534 107.86
107.19
549
115.65
113.25
564
119.13
117.85
579
118.50
118.31
594
123.34
122.73
535 103.95
102.83
550
117.20
116.28
565
120.46
119.69
580
114.90
115.02
595
122.36
122.22
536 107.25
106.56
551
118.69
117.79
566
115.93
114.64
581
111.00
110.02
596
119.95
119.95
537 105.58
104.63
552
115.55
113.57
567
112.61
111.46
582
111.97
111.49
597
113.42
112.81
538 102.80
102.34
553
111.45
110.12
568
113.85
113.23
583
111.47
110.35
598
102.65
102.75
539 103.95
103.66
554
109.32
108.18
569
116.79
116.52
584
106.64
105.83
599
107.57
108.17
540 103.96
103.92
555
107.96
106.83
570
116.29
114.85
585
112.75
112.19
600
111.28
111.05
127
Appendices
Appendices B: The average of daily power of load and demand of Sulaymaniyah/ directorate of control & communication in the January of 2013 until July of 2015 No. Load
Demand No.
Load
Demand No.
Load
Demand No.
Load
Demand No. Load
Demand
601 111.14
110.36
616
97.49
97.40
631
79.28
79.24
646
69.80
69.92
661
82.12
82.05
602 113.51
112.87
617
93.86
93.64
632
80.91
80.91
647
68.58
68.56
662
81.57
81.54
603 116.38
115.28
618
90.83
90.56
633
77.54
77.49
648
71.20
71.23
663
79.81
79.62
604 117.63
116.82
619
85.94
85.85
634
79.62
79.57
649
72.50
72.46
664
79.08
79.22
605 111.13
109.97
620
89.43
89.23
635
79.18
78.79
650
71.64
72.05
665
76.75
76.75
606 114.12
113.84
621
92.09
91.86
636
80.44
80.30
651
71.61
71.64
666
75.03
75.31
607 113.99
113.19
622
92.25
91.85
637
78.91
78.81
652
70.92
70.97
667
74.02
74.28
608 114.54
113.31
623
90.65
90.49
638
79.44
79.43
653
69.36
69.33
668
76.87
77.34
609 114.17
113.46
624
89.37
89.09
639
79.15
79.09
654
70.57
70.95
669
87.72
88.13
610 112.33
111.54
625
87.98
87.93
640
83.10
83.46
655
81.50
81.74
670
92.40
92.16
611 109.52
107.85
626
83.86
83.65
641
69.11
69.25
656
85.27
85.30
671
103.82
104.50
612
97.33
96.45
627
88.10
88.08
642
67.52
67.59
657
88.18
88.27
672
109.85
109.10
613 101.00
101.39
628
88.44
88.47
643
68.17
68.29
658
92.47
92.27
673
119.17
119.15
614 101.36
100.88
629
81.62
81.54
644
67.86
67.98
659
91.36
91.58
674
119.60
141.77
615
98.15
630
79.37
79.31
645
70.13
70.00
660
88.31
88.47
675
118.09
114.06
99.10
128
Appendices
Appendices B: The average of daily power of load and demand of Sulaymaniyah/ directorate of control & communication in the January of 2013 until July of 2015 No. Load
Demand No.
Load
Demand No.
Load
Demand No.
Load
Demand No. Load
Demand
676 111.65
108.21
691
123.21
119.71
706
127.75
123.45
721
150.66
148.76
736
150.88
146.80
677 110.13
107.15
692
124.58
121.46
707
125.54
122.57
722
153.39
148.09
737
157.18
156.59
678 110.73
108.61
693
127.14
125.57
708
129.73
127.65
723
149.60
145.17
738
156.08
158.95
679 108.27
106.25
694
128.62
125.93
709
130.79
126.24
724
144.18
139.72
739
160.10
156.08
680 107.34
106.30
695
134.17
132.03
710
123.85
120.83
725
147.23
145.16
740
164.49
160.30
681 105.06
104.21
696
134.34
129.37
711
126.90
125.30
726
149.22
143.25
741
164.39
161.10
682 102.05
101.57
697
133.95
130.00
712
135.44
132.98
727
145.28
141.39
742
162.75
159.66
683 100.36
98.94
698
132.31
128.86
713
140.66
139.14
728
142.17
137.50
743
160.58
157.28
684 101.25
98.85
699
132.72
131.49
714
140.22
135.93
729
139.07
135.87
744
157.48
153.24
685 110.41
111.13
700
139.47
138.28
715
139.77
136.03
730
141.22
139.76
745
152.42
147.86
686 114.89
112.47
701
141.08
136.64
716
141.49
136.96
731
140.37
135.58
746
151.31
146.65
687 112.06
109.61
702
136.04
130.11
717
137.47
134.70
732
140.42
137.72
747
153.59
151.02
688 110.44
109.00
703
129.42
125.35
718
142.63
139.86
733
147.98
147.12
748
154.05
150.01
689 115.35
110.72
704
128.78
125.11
719
144.04
139.92
734
151.30
146.23
749
153.31
148.79
690 122.20
122.78
705
129.68
126.58
720
144.52
141.32
735
148.75
146.15
750
151.79
148.27
129
Appendices
Appendices B: The average of daily power of load and demand of Sulaymaniyah/ directorate of control & communication in the January of 2013 until July of 2015 No. Load
Demand No.
Load
Demand No.
Load
Demand No.
Load
Demand No. Load
Demand
751 151.01
147.43
766
134.36
130.52
781
158.49
158.71
796
130.08
125.74
811
129.29
129.88
752 146.63
143.48
767
136.58
134.06
782
162.33
158.93
797
122.22
117.94
812
131.95
129.20
753 148.01
145.36
768
138.31
133.64
783
159.56
156.82
798
122.54
120.90
813
127.94
125.12
754 149.25
145.56
769
139.62
137.13
784
152.88
148.38
799
122.46
118.51
814
125.66
123.28
755 147.18
143.01
770
140.86
137.20
785
150.82
148.98
800
114.55
110.28
815
109.32
108.33
756 144.21
139.33
771
139.99
135.86
786
148.53
144.39
801
105.13
102.88
816
107.09
106.55
757 146.32
144.76
772
141.76
138.29
787
139.96
135.53
802
101.05
99.83
817
111.25
110.55
758 149.74
146.66
773
140.55
136.57
788
134.76
130.63
803
95.65
95.05
818
119.02
117.51
759 148.50
146.42
774
141.69
138.13
789
129.99
125.83
804
99.68
99.70
819
115.73
113.95
760 148.75
142.55
775
143.01
139.46
790
132.69
131.24
805
103.91
103.17
820
109.04
107.50
761 145.89
141.14
776
147.24
146.16
791
140.44
138.94
806
104.32
103.38
821
105.55
103.41
762 142.31
139.15
777
149.83
148.14
792
146.08
143.61
807
101.27
99.35
822
93.86
93.79
763 144.33
142.05
778
147.04
143.02
793
146.75
143.21
808
102.08
102.94
823
95.35
95.33
764 142.26
137.61
779
144.01
141.50
794
138.77
134.42
809
96.61
95.14
824
89.99
89.91
765 137.52
133.30
780
147.55
147.92
795
134.40
131.61
810
112.04
113.53
825
85.79
85.47
130
Appendices
Appendices B: The average of daily power of load and demand of Sulaymaniyah/ directorate of control & communication in the January of 2013 until July of 2015 No. Load
Demand No.
Load
Demand No.
Load
Demand No.
Load
Demand No. Load
Demand
826
82.20
81.94
841
75.85
75.35
856
72.90
72.70
871
87.29
86.98
886
101.21
99.51
827
79.11
78.75
842
85.82
86.19
857
67.86
67.66
872
86.04
85.64
887
99.62
98.16
828
74.46
74.43
843
91.27
88.98
858
72.19
71.97
873
84.84
84.52
888
101.84
100.52
829
69.83
69.76
844
96.26
94.33
859
72.61
72.09
874
86.47
86.29
889
104.46
103.12
830
76.46
76.55
845
91.35
87.65
860
73.16
73.00
875
86.99
86.77
890
101.64
99.34
831
76.77
76.69
846
83.78
81.50
861
74.14
73.77
876
88.04
87.48
891
93.84
92.10
832
95.85
96.37
847
78.76
76.82
862
76.19
76.07
877
86.03
86.08
892
95.78
95.59
833 107.23
106.11
848
75.35
74.55
863
75.45
75.27
878
87.27
87.15
893
101.25
100.14
834 103.84
102.56
849
75.67
74.79
864
70.50
70.47
879
90.75
90.86
894
101.23
99.09
835
95.03
93.71
850
68.51
68.48
865
72.37
72.25
880
91.26
90.11
895
100.23
98.78
836
81.55
81.48
851
72.19
72.19
866
78.28
78.06
881
95.51
94.41
896
101.59
100.53
837
83.32
83.03
852
71.26
71.08
867
83.12
82.85
882
103.75
102.46
897
101.54
100.46
838
79.70
79.57
853
74.02
73.91
868
87.02
86.85
883
104.66
102.21
898
99.89
98.12
839
79.17
79.15
854
73.56
73.44
869
89.24
89.00
884
98.52
97.15
899
102.89
101.40
840
78.64
78.54
855
72.25
71.97
870
86.46
86.00
885
102.01
101.54
900
103.71
101.65
131
Appendices
Appendices B: The average of daily power of load and demand of Sulaymaniyah/ directorate of control & communication in the January of 2013 until July of 2015 No. Load
Demand No.
Load
Demand No.
Load
Demand
901 104.30
102.49
916
112.82
110.15
931
107.02
105.91
902 105.74
104.11
917
113.14
111.78
932
107.56
106.02
903 108.68
107.34
918
114.94
114.09
933
105.46
103.77
904 105.49
103.32
919
109.99
107.71
934
109.75
110.36
905
95.66
94.14
920
111.36
109.96
935
114.34
113.68
906
96.02
95.75
921
114.98
114.04
936
115.44
114.05
907
97.89
97.96
922
116.45
114.78
937
113.78
112.74
908 101.98
101.71
923
113.04
110.72
938
117.45
116.90
909 103.44
102.36
924
115.45
113.98
939
119.00
118.96
910 102.85
101.12
925
117.50
117.91
940
116.07
113.46
911 109.42
109.36
926
114.90
113.49
912 109.50
108.60
927
103.91
103.14
913 112.71
112.33
928
100.94
99.88
914 117.58
116.17
929
101.90
100.87
915 114.49
111.81
930
103.87
103.19
132
@ @˜ƒÝ¾a
@@
@æà@b"@b¾@Š†b—¾a@óïÕi@µi@æà@òŠa‡—Üa@ÊÔíà@o¥@ônÜa@óÔb Üa@Ëaíäc@âèc@õ‡yg@ðè@óï÷bi‹éÙÜa@óÔb Üa@çg @ðÈbå—Üa@Ëb ÕÜa@Þrà@~óÑÝnƒ¾a@pbÈb ÕÜa@óïáåm@À@âéà@Šì†@b"@béäaì@báØ@óïàíïÜa@båmbïy@À@âéà@*qdm @òa†c@ðèì@çb;äýa@òbï¢@‹’bj¾a@*Ì@ìc@@‹’bj¾a@ÞÙ“i@Âjm‹m@CÜa@pbÈb ÕÜa@æà@@Ëb Ô@@ña@ìc@ñ†b—nÔýaì @ @Nòbï¨a@À@óîŠa‹ánGýa@ð Ém@CÜa@pbàbé¾a@æà@*rØ@Œb−fi @ì@ Xi@óï÷bi‹éÙÜa@óÔb @ôÝÈ@kÝ Üa@I@báè@óï÷bi‹éÙÜa@óÔb ÝÜ@pbäbïjÜa@æà@µÈíä@oà‡ƒnGg@óGaŠ‡Üa@ë‰è@À @öbåi@íè@óÜbG‹Üa@ë‰è@À@ðGbGýa@Ó‡"aì@óïäbáïÝ;Üa@óÅÐb«@À@~HYi@@ðàíïÜa@ÞÙ“i@óÔb Üa@ÛþénGg @CÜa@ÞïÝznÜa@xˆbáåÜa@æà@‡î‡u@¹@ãa‡ƒnGhi@Ú܈ì@óï÷bi‹éÙÜa@óÔb Üa@ßíy@pbäbïjÝÜ@ðÝïÝznÜa@xˆíáåÜa @óÔb @ßíy@pbäbïjÜa@æà@ËíåÜa@a‰è@À@@‡uaínm@CÜa@ò‡Õɾa@ÛíÝ;Üa@öa‡ig@@ì@ŠbéÄg@À@@óïÜbÈ@@òöbÑØ@b" @æáš@óïGb;¨aì@ëöbÑÙÜa@æà@ðÜbÈ@õín;à@paˆì@da‡u@‘b;¨a@xˆíáåÜa@í@è@xˆíáåÜa@a‰è@çaì@óï÷bi‹éÙÜa @ @ .RNN HòŠ‹Ùn¾a@óïj—ÉÜa@pbÙj“ÜaI@xˆíáåÜbi@ôá;m@CÜaì@óÉå —¾a@óïj—ÉÜa@pbÙj“Üa @CÜa@@ò‡ï§a@w÷bnåÜa@æà@ðÜbÈ@õín@;à@p‹éÄc@óGaŠ‡Üa@ë‰è@À@oà‡ƒ@nGa@CÜa@xˆbáåÜa@æà@ËíåÜa@a‰è@À @ @N@xˆbáåÜa@æà@õ‹‚ýa@Ãb¹ýc@æÈ@bèiï¸ @Ëaíäc@óÐbØ@Êà@ÞàbÉnm@béäa@ðè@óïGbGýa@bémaiïà@õ‡yg@@òŠ‹Ùn¾a@óïj—ÉÜa@pbÙj“Üa@xˆíáåÜa@çg@ãbÈ@ÞÙ“i @@I@óÜiÜin¾a@L@H@ChaoticI@ó@îíšíÑÜa@ÛíÝG@Þrà@ò‡ÕÉà@ÛíÝGì@Ãb¹a@†íuíi@Œbn¸@CÜa@pbäbïjÜa @xˆíáåÜa@æà@ËíåÜa@a‰è@ça@báØ@H@çìapÜa@óØ‹yI@l@çíîbîiïÑÜa@béïá;î@ñ‰Üa@pbÑ—Üa@oÉi@ìc@~@HSeismic @a‰éi@iïánm@@CÜa@pbäbïjÜa@æà@@óïåàiÜa@óÝ;ÝG@paˆ@@óï ©a@*Ì@pbÑ–@paˆ@pbäbïjÜa@Êà@âv;åm@ì@ÞàbÉnm @ @NóÑ—Üa
@óïj—ÉÜa@óÙj“Üa@xˆbáåÜa@óïÕji@bénäŠbÕà@æÙá¾a@æà@HRNNsI@òŠ‹Ùn¾a@óïj—ÉÜa@pbÙj“Üa@@xˆí¹@çc@báØ @†‡¤ì@óïу¾a@pbÕj @Þ‚a†@ò†íuí¾a@Hnodes@I@pa‡ÕÉÜa@À@pa*Ím@t‡¥@béäc@sïy@HFFNNI@Þrà @óïàŒŠaí‚@ãa‡ƒnGhi@Ú܈ì@Hupdating hidden nodes weights@I@pa‡ÕÉ@Üa@ë‰è@Þ‚a†@béäaŒìc @ @Nóïj—ÉÜa@pbÙj’@xˆbáåÜa@À@㇃n;m@ônÜa@pbïàŒŠaí©a@óïÕjÜa@æÈ@ÒÝn¦@‡î‡vnÜa@a‰"@ó——ƒn¾a @ßìýa@Z@µîín;¾a@ôÝÈ@óï÷bi‹éÙÜa@óÔb Ü@@HRNNI@xˆí¹@Þ›Ðc@@†b°hi@@sybjÜa@ãbÔ@óÜbG‹Üa@ë‰è@À @~@@Hmodel generalizationI@@Š‡Õ¾a@xˆíáåÜa@âïáÉm@òíÔ@‡î‡znÜ@HPredictionI@@µáƒnÜ@xˆí¹ @ì@@HR2I@ŠbïÉà@ãa‡ƒnGg@db Åîa@ì@HAICI@ì@HMSEI@Þrà@@‹î‡ÕnÜa@d ©@óÔ‡Üa@~îb Õà@ãa‡ƒnGhi@Ú܈ì @kÝ @õín;à@‘bGc@ôÝÈ@üjånÝÜ@@@H1-5-10-1@HRNNI@I@Õ¾a@xˆíáåÜa@ãa‡ƒnGg@íè@ðäbrÜa@õín;¾a @o¸@CÜa@w÷bnåÜa@âèc@æà@pnÉm@CÜa@óïåàiÜa@óÝ;ÝG@ë‰è@êïÝjÕn;¾a@ì@êïÜb¨a@bèŠbÉGhi@óï÷bi‹éÙÜa@óÔb Üa @òöbÑØ@paˆ@@@HRNNI@xˆíáåÜa@çhi@ßíÕÜa@çbÙàfibi@w÷bnåÜa@ë‰è@öíš@ôÝÈ@ì@óGaŠ‡Üa@ë‰è@À@bé bjånGg @ò‹Øaˆ@†íuì@íè@xˆíáåÜa@a‰è@pbÑ—Üa@õ‡yg@æà@ça@íè@báØ@ò‡Õɾa@@Ãb¹ýa@æá›nm@CÜa@ÞGþ;Üa@ÞÙÜ@óïÜbÈ @CÜaì@xˆbáåÜa@óïÕi@æÈ@ëiïº@bà@ć
b Åîa@a‰èì@ßí c@ò‡¾@@pbàíÝɾa@çi¦@ça@b"@CÜa@óïу¾a@pbÕj Üa@À @ @N@óïåàiÜa@óÝ;Ý;Üa@pbäbïjÜa@ÞïÝznÜa@À@béåà@ò†bÑnGfia@æÙº @CÜa@pbïàŒŠaí©a@µi@óÑrÙà@ì@óÕïÔ†@óäŠbÕà@íè@óÜbG‹Üa@ë‰è@À@ëŒb−g@@CÜa@w÷bnåÜa@âèc@õ‡yc@çcì@báØ @Öî‹ @æÈ@ßba@ë‰è@À@öap©a@öaŠdi@@óäbÉnGýbi@ì@béåà@‡î‡ÉÜa@ói‹¤@‡Éi@Ú܈ì@㇃n;m@‡ Ô@ìa@㇃n;m Levenberg-@I@óïàŒŠaí©a@çdi@p‹éÄ@óà‡ƒn;¾a@âéÝ÷bGŠì@óïáÝÉÜa@Š†b—¾@bi@sybjÜa@óäbÉnGg @çaŒìýa@pbÐíÑ—à@‹î‡ÕmìóÙj“Üa@km@óïäbÙàgì@óÈ‹;Üa@sïy@æà@pbïàŒŠaí©a@Þ›Ða@ðè@H@ Marquardt @ôÝÈ@ßí—¨a@@óvïnåÜa@À@ì@pbÙj“Üa@âÝÉnÜa@À@㇃n;m@CÜa@pbïàŒŠaí©a@óïÕji@dbnäŠbÕà@öíÑÙÜa@ÞÙ“i @kÝ Üa@oåá›m@CÜa@óGaŠ‡Üa@‡ïÔ@óïåàiÜa@óÝ;ÝG@Þ‚a†@ò†íuí¾a@Ãb¹ŁÜ@óibvnGfia@À@óïäbÙàg@paˆ@xˆí¹ @À@ðÕïj nÜa@käb§a@À@bénGaŠ†@o¸@CÜa@ì@‹Ø‰Üa@óÕibG@óïäbáïÝ;Üa@óå@óï÷bi‹éÙÜa@óÔb Ü@@ÛþénGfiaì NóGaŠ‡Üa@ë‰è
@@
ón‚íq
@@
@ìbšŠói@ìòŠìó @ôÙŽïÜûŠ@bèòìŠóè@a‡ïmóîbÄû‹à@ôäbîˆ@óÜ@çbØóïØòŠó @óïn îìa‡Žïq@æî#$ä‹ @óÜ@óÙŽïØóî@biòŠbØ@ñòŒìì @óäaˆûŠ@óØ@çbîˆ@ôäbØòŠaíi@óÜ@ÚŽîŠaíi@Šóè@ì@õŒb ó“ïq@ì@õŠìíib÷@õŠaíi@íØòì@a‡äbØbïubïu@òŠaíi@ôån ƒ“Žïq@óÜ@oŽïåïiò† @ @Nôä‡äbîó @ãb−ó÷@ói@oŽïn ò‡Üóè@Äû‹à @LbiòŠbØ@ õòŒìì@ ôn aí‚@ I@ óÜ@ µnî‹i@ ”ïäaìó÷@ òìa‹åŽïèŠbØói@ biòŠbØ@ õòŒìì@ ôn îìa‡Žïq@ óÜ@ Šüu@ ìì†@ a†óîóàbä@ ãóÜ @@ÚŽïÝŽî†üà@ô䆋Ø@o ì슆@óÜ@óïnî‹i@óîóàbä@ãó÷@ôØòŠó @ ð−bàb÷@NðäbáŽïÝ @ñb >ŽîŠbq@óÜ@HbiòŠbØ@õòŒìì@ôäbåŽïèŠbØói @ðØóîbäaím@óØ@òìó䆋Ùï’@ðÝŽî†üà@@ôŽîíä@ôÙŽîŒaíŽï’@ôäbåŽïèŠbØói@ói@”îìó÷@biòŠbØ@õòŒìì@Šó @óÜ@ñŠbïäaŒ@õŠbÙï’@üi @oŽîìóØò†@ìbšŠói@biòŠbØ@õòŒìì@Šó @óÜ@óäbîŠbïäaŒ@òŠüu@ãóÜ@óØ@ñóäaŒüÜb÷@ón’ìòŠ@ìŠüu@ìó÷@@n ‚Šò†@óÜ@óîóè@ñŒŠói Artificial Neural I@ çbØò†‹Ùn ò†@ òŠbàò†@ òŠüm@ óÜ@ ŒŠói@ o b÷@ ì@ Šbïn óè@ ŠûŒ@ ôÙŽîŠüu@ óÜ@ óïnî‹i@ ”îìó÷ @óÝŽî†üà@ òŠüu@ ãó÷@ HRecurrent Neural NetworkI@ òìaŠó @ ñŠbàò†@ õŠüm@ ói@ oŽî‹iò†@ ìbä@ óØ@ HNetwork @ói@NòìómbØò†@ñbïu@õ‹m@ôäbØóÝŽî†üà@óÜ@ôäbØó’bi@óàb−ó÷@ì@ñŒŠói@o b÷@a†óîóàbä@ãóÜ@òìímbèŠbØói@õóïîŠó òŒbm @óØ@ pbØò†@ ÛóïîŠbïäaŒ@ ì@ bma†@ òŠüu@ ìíàóè@Žßó óÜ@ pìóØí Üóè@óØ@ óîa†òìóÜ@ óÝŽî†üà@ òŠüu@ ãó÷@ ð’bi@ ôn“ @ ôØóîòíŽï’ @@ðmbÑï @óÜ@ñü‚@•óäbmbÑï @òŠüu@ãó÷@óØ@ oŽî‹Øò†@ñ†ói@a‡äbïäbØbma†@óÜ HComplex behaviorsI@ŒüÜb÷@ômbÑï @çìa‹i@õóÜíu@æŽïÜò†@ðŽïš@çbØóïîbî>ïÐ@óØ@ñóäaŒüÜb÷@ómbÑï @ìóÜ@ÚŽîŠüu@çbî @HseismicI@~@HChoaticI@ìaˆòìaˆ @ðÝŽïèbä@ ðmóÑï @ óØ@ oŽïåŽï−í ò†@ ñü‚@ óäbîbma†@ òH−Œ@ ìó÷@ Žßó óÜ@ óÝŽî†üà@ òŠüu@ ãó÷@ pbØìbè@ òìónŽïåïiò†@ HBMI @ÚŽîŠüu@ ói@ óîa‡Žïm@ çbîóïïÝŽïèbä@ ðmóÑï @ ãó÷@ pbØ@ ñòH−Œ@ ðäbØbma†@ ìíàóè@ ”ïn“ @ ói@ ìóîa‡Žïm HnonlinearI @ìbä@ðäbØò†üä@óÜ@ñŠbÙäaŠü @óØ@ñòìói@HFFNNI@Ûòì@çbØòŠbàò†@òŠüm@ñ‹m@ðäbØóÝŽî†üà@óÜ@Œaìbïu@pbØò†@póØí Üóè Updating hidden nodes I@pbØò†@a†óäa†üä@ãó÷@ðäbØó“ŽïØ@óÜ@ñŠó òŒbm@ãaìò†Šói@ói@ì@pbØò†@çbØòìaŠb’@óåïš @ @NóÝŽî†üà@òŠüu@ãó÷@üi@pójîbm@ðÙŽïánîŠü$Üó÷@ðäbåŽïèŠbØói@ói@Hweight
@çbïàóØóî@ o b÷@ ìì†@ Šó óÜ@ biòŠbØ@ õòŒìì@ üi@HRNNI@ ÚŽïÝŽî†üà@ æî#’bi@ õòìóåîŒû†@ ói@ bn óè@ Šò‰Žîím@ a†óîóàbä@ ãóÜ @óÜóè@ðä‡äb$äó Üóè@ñŠòíŽïq@ói@óØóÝŽî†üà@ñòìóäbà@ñ>Žïè@ô䆋ÙîŠbî†@üi@HPredictionI@ç‡äb$äó Üóè@üi@ÚŽïÝŽî†üà @a‡àòìì†@ K b÷@ óÜ@ Šbu@ aì†@ HR2I@ 熋ÙîŠbî†@ ñóÙÜüØìbè@ bèòìŠóè@ HAICI@ bèòìŠóè@ HMSEI@ Ûòì@ HErrorI @ói@ biòŠbØ@ õòŒìì@ õŒaí‚a†@ ôn b÷@ Šó @ óÜ@ 熋Ø@ ôåïj“Žïq@ üi@ H1M5M10M1I@ ìa‹ØŠbïå“Žïq@ ðÝŽî†üà@ ðäbåŽïèŠbØói @a†óîóàbä@ãóÜ@òìímbè@o ò†ói@óØ@ãb−ó÷Šò†@æî#$ä‹ @óÜ@óïnî‹i@•óàó÷@òìóîòH−Œ@ãó÷@õìímbèa†@ì@bn Žï÷@ðäbØó‚‹ä @óÜ@ óØ@ õóäaH−Œ@ ìó÷@ ìíàóè@ üi@ óîóè@ õŒŠói@ ôØóîbäaím@ HRNNI@ ôÝŽî†üà@ óØ@ µŽïÝi@ µäaímò†@ •óàb−ó÷Šò†@ ãóiòì @HRNNI@óÜ@ óÝŽî†üà@òŠüu@ãó÷@ô’bi@ôÙŽïmóÑï @bèòìŠóè@òì@óîóè@HPatternsI@ŒüÜb÷@ ñŒaíŽï’@ôn’ìòŠ@a‡ïäbØómbéÙŽïq @üi@çbØóîŠbïäaŒ@óîóè@a‡îbäaím@óÜ@óØ@óîóè@HMemoryI@ Hi@HHidden LayerI@a†òìaŠb’@@@ïšóÜ@ óØ@óîa†òìóÜ @ @Na†bma†@ñŠbÙï’@ðmbØ@óÜ@ó’bi@ðÙŽî†í @•óàó÷@óØ@oŽî‹$jÜóè@‹mbîŒ@ôØóîòìbà @ñòìó䆋ÙïÔbm@ ñaì†@ a‡äbØóánîŠü$Üb÷@ çaíŽïä@ óÜ@ ‹š@ ì@ †Šìì@ ôَìaŠói@ óÜ@ óïnî‹i@ óîóàbä@ ãó÷@ ñ‹m@ ðÙŽïàb−ó÷Šò† @pìóØŠò†@òìóäbïäbØóàbä@ìòìbšŠó @ñó$ŽîŠ@óÜ@òŠaíi@ãó÷@ðäbîaŒòŠb’@ói@´ ói@o“q@ói@ì@çbØóánîŠü$Üó÷@óÜ@ŠûŒ@ðÙŽï’ói @çìíi‹ŽïÐ@ ñó$ŽîŠ@ óÜ@ ‹m@ ðäaìóÜ@ ìíi@ æî‹m@ bäaím@ ói@ ì@ æî#’bi@ HLevenberg-MaequardtI@ ðánîŠü$Üó÷@ óØ @æî‹mŒŠói@ì@pbØ@æî#àóØ@ói@HRNNI@çbØòìaŠó @óïîŠbàò†@ òŠüm@üi@HTrainingI@熋َïq@Ö’óà@ì@HLearningI @æî#’bi@ óØ@ båŽïè@ o ò†@ ói@ çb¹bØóîŠbàò†@ òŠüm@ ðÙŽïÝŽî†üà@ a‡àb−ó÷@ óÜ@ ‹m@ ðäbØóánîŠü$Üó÷@ Žßó óÜ@ †ŠìaŠóiói@ bäaím @ì@ o aí‚I@ biòŠbØ@ õòŒìì@ õòH−Œ@ ðäbØòŒüÜb÷@ ón’ìòŠ@ Žßó óÜ@ óîóè@ HBest PerformanceI@ õòìóäa‡àłòì @ @@Na†óîóàbä@ãóÜ@ðÙïnØa‹q@ðäóîýóÜ@òìa‹Ø@ó aŠ†@ìòìa‹ÙŽïq@õòˆbàb÷@óØ@ðäbáŽïÝ @ñŠb’@üi@HçbåŽïèŠbØói
