Selection of Techniques and Metrics Raj Jain Washington University in Saint Louis Saint Louis, MO 63130 [email protected] These slides are available on-line at: http://www.cse.wustl.edu/~jain/cse567-06/ Washington University in St. Louis

CSE567M

3-1

©2006 Raj Jain

Overview Criteria for Selecting an Evaluation Technique ! Three Rules of Validation ! Selecting Performance Metrics ! Commonly Used Performance Metrics ! Utility Classification of Metrics ! Setting Performance Requirements !

Washington University in St. Louis

CSE567M

3-2

©2006 Raj Jain

Criteria for Selecting an Evaluation Technique

Washington University in St. Louis

CSE567M

3-3

©2006 Raj Jain

Three Rules of Validation !

Do not trust the results of a simulation model until they have been validated by analytical modeling or measurements.

!

Do not trust the results of an analytical model until they have been validated by a simulation model or measurements.

!

Do not trust the results of a measurement until they have been validated by simulation or analytical modeling.

Washington University in St. Louis

CSE567M

3-4

©2006 Raj Jain

Selecting Performance Metrics

Washington University in St. Louis

CSE567M

3-5

©2006 Raj Jain

Selecting Metrics !

!

!

Include: " Performance Time, Rate, Resource " Error rate, probability " Time to failure and duration Consider including: " Mean and variance " Individual and Global Selection Criteria: " Low-variability " Non-redundancy " Completeness

Washington University in St. Louis

CSE567M

3-6

©2006 Raj Jain

Case Study: Two Congestion Control Algorithms Service: Send packets from specified source to specified destination in order. ! Possible outcomes: " Some packets are delivered in order to the correct destination. " Some packets are delivered out-of-order to the destination. " Some packets are delivered more than once (duplicates). " Some packets are dropped on the way (lost packets). !

Washington University in St. Louis

CSE567M

3-7

©2006 Raj Jain

Case Study (Cont) !

Performance: For packets delivered in order, " Time-rate-resource ⇒ ! Response time to deliver the packets ! Throughput: the number of packets per unit of time. ! Processor time per packet on the source end system. ! Processor time per packet on the destination end systems. ! Processor time per packet on the intermediate systems. " Variability of the response time ⇒ Retransmissions ! Response time: the delay inside the network

Washington University in St. Louis

CSE567M

3-8

©2006 Raj Jain

Case Study (Cont) Out-of-order packets consume buffers ⇒ Probability of out-of-order arrivals. " Duplicate packets consume the network resources ⇒ Probability of duplicate packets " Lost packets require retransmission ⇒ Probability of lost packets " Too much loss cause disconnection ⇒ Probability of disconnect "

Washington University in St. Louis

CSE567M

3-9

©2006 Raj Jain

Case Study (Cont) !

Shared Resource ⇒ Fairness

!

Fairness Index Properties: " Always lies between 0 and 1. " Equal throughput ⇒ Fairness =1. " If k of n receive x and n-k users receive zero throughput: the fairness index is k/n.

Washington University in St. Louis

CSE567M

3-10

©2006 Raj Jain

Case Study (Cont) "

Throughput and delay were found redundant ⇒ Use Power.

Variance in response time redundant with the probability of duplication and the probability of disconnection " Total nine metrics. "

Washington University in St. Louis

CSE567M

3-11

©2006 Raj Jain

Commonly Used Performance Metrics !

Response time and Reaction time

Washington University in St. Louis

CSE567M

3-12

©2006 Raj Jain

Response Time (Cont)

Washington University in St. Louis

CSE567M

3-13

©2006 Raj Jain

Capacity

Washington University in St. Louis

CSE567M

3-14

©2006 Raj Jain

Common Performance Metrics (Cont) Nominal Capacity: Maximum achievable throughput under ideal workload conditions. E.g., bandwidth in bits per second. The response time at maximum throughput is too high. ! Usable capacity: Maximum throughput achievable without exceeding a pre-specified response-time limit !

!

Knee Capacity: Knee = Low response time and High throughput

Washington University in St. Louis

CSE567M

3-15

©2006 Raj Jain

Common Performance Metrics (cont) ! ! !

Turnaround time = the time between the submission of a batch job and the completion of its output. Stretch Factor: The ratio of the response time with multiprogramming to that without multiprogramming. Throughput: Rate (requests per unit of time) Examples: " Jobs per second " Requests per second " Millions of Instructions Per Second (MIPS) " Millions of Floating Point Operations Per Second (MFLOPS) " Packets Per Second (PPS) " Bits per second (bps) " Transactions Per Second (TPS)

Washington University in St. Louis

CSE567M

3-16

©2006 Raj Jain

Common Performance Metrics (Cont) !

!

Efficiency: Ratio usable capacity to nominal capacity. Or, the ratio of the performance of an n-processor system to that of a one-processor system is its efficiency. Utilization: The fraction of time the resource is busy servicing requests. Average fraction used for memory.

Washington University in St. Louis

CSE567M

3-17

©2006 Raj Jain

Common Performance Metrics (Cont) Reliability: " Probability of errors " Mean time between errors (error-free seconds). ! Availability: " Mean Time to Failure (MTTF) " Mean Time to Repair (MTTR) " MTTF/(MTTF+MTTR) !

Washington University in St. Louis

CSE567M

3-18

©2006 Raj Jain

Utility Classification of Metrics

Washington University in St. Louis

CSE567M

3-19

©2006 Raj Jain

Setting Performance Requirements !

!

Examples: “ The system should be both processing and memory efficient. It should not create excessive overhead” “ There should be an extremely low probability that the network will duplicate a packet, deliver a packet to the wrong destination, or change the data in a packet.” Problems: Non-Specific Non-Measurable Non-Acceptable Non-Realizable Non-Thorough ⇒ SMART

Washington University in St. Louis

CSE567M

3-20

©2006 Raj Jain

Case Study 3.2: Local Area Networks ! !

! !

!

Service: Send frame to D Outcomes: " Frame is correctly delivered to D " Incorrectly delivered " Not delivered at all Requirements: Speed " The access delay at any station should be less than one second. " Sustained throughput must be at least 80 Mbits/sec. Reliability: Five different error modes. " Different amount of damage " Different level of acceptability.

Washington University in St. Louis

CSE567M

3-21

©2006 Raj Jain

Case Study (Cont) " " " "

" "

The probability of any bit being in error must be less than 1E-7. The probability of any frame being in error (with error indication set) must be less than 1%. The probability of a frame in error being delivered without error indication must be less than 1E-15. The probability of a frame being misdelivered due to an undetected error in the destination address must be less than 1E-18. The probability of a frame being delivered more than once (duplicate) must be less than 1E-5. The probability of losing a frame on the LAN (due to all sorts of errors) must be less than 1%.

Washington University in St. Louis

CSE567M

3-22

©2006 Raj Jain

Case Study (Cont) !

Availability: Two fault modes – Network reinitializations and permanent failures " The mean time to initialize the LAN must be less than 15 milliseconds. " The mean time between LAN initializations must be at least one minute. " The mean time to repair a LAN must be less than one hour. (LAN partitions may be operational during this period.) " The mean time between LAN partitioning must be at least one-half a week.

Washington University in St. Louis

CSE567M

3-23

©2006 Raj Jain

Summary of Part I

!

!

Systematic Approach: Define the system, list its services, metrics, parameters, decide factors, evaluation technique, workload, experimental design, analyze the data, and present results Selecting Evaluation Technique: The life-cycle stage is the key. Other considerations are: time available, tools available, accuracy required, trade-offs to be evaluated, cost, and saleability of results.

Washington University in St. Louis

CSE567M

3-24

©2006 Raj Jain

Summary (Cont) Selecting Metrics: " For each service list time, rate, and resource consumption " For each undesirable outcome, measure the frequency and duration of the outcome " Check for low-variability, non-redundancy, and completeness. ! Performance requirements: Should be SMART. Specific, measurable, acceptable, realizable, and thorough. !

Washington University in St. Louis

CSE567M

3-25

©2006 Raj Jain

Exercise 3.1 What methodology would you choose: a. To select a personal computer for yourself? b. To select 1000 workstations for your company? c. To compare two spread sheet packages? d. To compare two data-flow architectures, if the answer was required: i. Yesterday? ii. Next quarter? iii. Next year? Washington University in St. Louis

CSE567M

3-26

©2006 Raj Jain

Homework #2 Read chapters 3 ! Submit answers to " Exercise 3.1 !

Washington University in St. Louis

CSE567M

3-28

©2006 Raj Jain

Selection of Techniques and Metrics - Washington University in St ...

Performance Time, Rate, Resource. ➢ Error rate, probability. ➢ Time to failure and duration. ❑ Consider including: ➢ Mean and variance. ➢ Individual and Global.

464KB Sizes 0 Downloads 283 Views

Recommend Documents

Selection of Techniques and Metrics - Washington University in St ...
Case Study: Two Congestion Control Algorithms. ❑ Service: ... Some packets are delivered out-of-order to the ... Some packets are dropped on the way (lost.

The Crisis in Rural Primary Care - University of Washington
backbone of rural health care, yet primary care in rural loca- tions is in crisis. The number of students choosing primary care careers has declined precipitously.

Vector Symbolic Architectures - Washington and Lee University
We provide an overview of Vector Symbolic Architectures (VSA), a class ... Perhaps more so than any other sub-field of computer science, artificial intelligence .... The introduction of noise requires that the unbinding process employ a “cleanup.

University of Washington .pdf
Retrying... University of Washington .pdf. University of Washington .pdf. Open. Extract. Open with. Sign In. Main menu. Displaying University of Washington .pdf.

Winter Dreams - Washington State University
Dexter Green's father owned the second best grocery-store in. Black Bear--the best ... tramped with his eyes squinted up against the hard ..... "Better thank the Lord she doesn't drive a swifter ball," said Mr. ..... president of a great trust compan

PhD Opportunity: University of St Andrews and The ... -
PhD Opportunity: University of St Andrews and The James Hutton Institute, UK. Designing a Diagnostic Tool for the Detection of Viral Diseases in Agricultural ...

Postdoc in Statistical Ecology - Washington State University, Dr. Leslie ...
... academic transcripts and the names and contact details of three references to [email protected] by 1 June 2015. Start date will be as soon as possible after ...

Introduction to Wireless Signal Propagation - Washington University in ...
Note: This is the 2nd in a series of 2 lectures on wireless physical layer. Modulation ... Wireless Radio Channel. ❑ Path loss: .... Advantages of OFDM. ❑ Easy to ...

Postdoc in Statistical Ecology - Washington State University, Dr. Leslie ...
Position – Postdoc in Statistical Ecology - Washington State University, Dr. Leslie ... should be proficient with R; those with a background in ecological statistics,.

PACURH OTM Writing Guide - Central Washington University
You want to recognize them, but don't know the best way to go about it. Well, how ... There are two types of OTMs: General and Programs. Each has its own word ...

UW NW CSC Fellows RFP 2015 - University of Washington
delivery for managers and policy makers in the Northwest region ... research and outreach on issues that inform management decisions and public policy ... proposed research is required to be documented with an annual report, due by.

UW NW CSC Fellows RFP 2015 - University of Washington
A Cooperative Program of the ... for funding graduate research assistantships (NW CSC Fellows) at the MS or PhD level in ... research and outreach on issues that inform management decisions and public policy development related to climate.

PACURH OTM Writing Guide - Central Washington University
made a positive, meaningful impact during the month for which it is submitted” ... Acceptable Nominees: Senior staff advisors to a residential committee, college council, college ... In recent years, this OTM category has come to include passive pr

CALCULUS I: Math 124 H - University of Washington
Email: [email protected]. Office: Padelford C-339. Web page: ... Make sure to log onto webassign as soon as possible and attempt the first ... A link to the class website can be found at: http://www.math.washington.edu/∼aloveles/ You wil

CALCULUS I: Math 124 H - University of Washington
Email: [email protected]. Office: Padelford C-339. Web page: ... Make sure to log onto webassign as soon as possible and attempt the first ... A link to the class website can be found at: http://www.math.washington.edu/∼aloveles/ You wil

The Hardware/Software Interface Overview - University of Washington
Design was expensive the instruction set was very simple. E.g., a single ... Hardware rchitecture Specification (Interface). University of Washington. CSE351 ...

pdf-1836\history-of-the-royal-society-washington-university-studies ...
Connect more apps... Try one of the apps below to open or edit this item. pdf-1836\history-of-the-royal-society-washington-university-studies-by-thomas-sprat.pdf.

The Selection of High-Skilled EmigrantsMatthias Parey: University of ...
Oct 3, 2016 - London School of Economics, [email protected], Nicolai Netz: DZHW, [email protected]. We thank .... 3Since many papers investigate migrant selection between two countries only (see Online Ap- ..... and are considered comparable to a

manual-handbook-ripping-cat-selection-techniques-applications ...
www.CAT.com. Table of Contents. OF RIPPING. Page 3 of 32. manual-handbook-ripping-cat-selection-techniques-applications-production-compatibility.pdf.

Dynamic Adverse Selection - Economics - Northwestern University
Apr 14, 2013 - capturing our main idea that illiquidity may separate high and low quality assets in markets ... that she might later have to sell it, the owner of an asset had an incentive to learn its quality. ..... The proof is in an online appendi