Distributed Node with Distributed Quota System (DNDQS) Author: Eric Yu ([email protected]) Date: 2016/02/23

Contents • • • • • • •

Scenario Assumption Architectures Roles of components Characteristics & Benefits Potential Problems Improvements

Scenario • A Distributed node system would like to apply API usage quota for some query service across all nodes. • We only allow nodes to call the query service for some certain amount of quota. No matter the call is come from which node, it just counts. • After use out the quota, all further query service would be blocked.

Assumptions • Distributed node system which allow dynamic node insert/delete. • Nodes are behind load balancing. Perfect load balance is not required. • Small and under-controlled number of false positive quota is acceptable. • Nodes are able to reach each others though network.

Architecture - Distributed node

Node 1

Node 2

Node 3 Node 4

With Consistent hashing

Node 1

Node 2

Node 3 Node 4

Consistent Hashing

With Consistent hashing – Delete node

Node 2

Node 1

Node 3

Consistent Hashing (Deleting Node 4)

With Consistent hashing – Insert node Node 5 Node 1

Node 2

Node 3 Node 4

Consistent Hashing (Inserting Node 5)

Quota Server – sync quota to nodes

Node 1

Node 2

Quota Server Quota: 1000 Node 3 Node 4

Quota Server – sync quota to nodes Quota: 250 Quota: 250

Node 1

Node 2

Quota Server Quota: 1000 Node 3 Node 4

Quota: 250

Quota: 250

Node quota negotiation Quota: 180 Quota: 120

Node 1

Node 2

Quota Server Quota: 1000 Node 3 Node 4

Quota: 150

Quota: 90

After some time ……

Node quota negotiation Quota: 180 Quota: 120

Node 1

Node 2

Quota Server Quota: 1000 Node 3 Quota: 90

Node 4

Periodic Checking Quota:

I have 90

Quota: 150 Give you 30

(150+90)/2 = 120 150-30 = 90+30 = 120

Node 3 Quota <100 => Request next peer (Node 4) share quota

Node quota negotiation Quota: 180 Quota: 120

Node 1

Node 2

Quota Server Quota: 1000 Node 3 Node 4

Quota: 120

Quota: 120

Node quota negotiation Quota: 180 Quota: 120

Node 1

Node 2

Quota Server Quota: 1000 Node 3 Node 4

Quota: 50

Quota: 120

After some time ……

Node quota negotiation Quota: 180 Quota: 120

Node 1

Node 2

Quota Server Quota: 1000

I have 50 Give you 35

Node 3 Quota: 120

Node 4 Quota: 50

(120+50)/2 = 85 120-35 = 50+35 = 85

Node quota negotiation Quota: 180 Quota: 85

Node 1

Node 2

Quota Server Quota: 1000 Node 3 Node 4

Quota: 85

Quota: 120

Node quota negotiation Quota: 0 Quota: 5

Node 1

Node 2

Quota Server Quota: 1000 Node 3 Node 4

Quota: 10

Quota: 15

After some time ……

Node quota negotiation Quota: 0 Quota: 5

Node 1

Node 2

Quota Server Quota: 1000

I have 10 No quota for share

Node 3 Node 4

Quota: 15

Quota: 10

When self Quota < 10, don't share quota with preceeding peer.

Node quota negotiation Quota: 0 Quota: 5

Node 1

Node 2

Quota Server Quota: 1000 Node 3 Node 4

Quota: 10

Quota: 15

Remain no change

Node stored information Node X

Stored information: - node list (node 1, node 2, node 3, node 4, ...) - local quota

• DO know who is the next node along the node chain.

Roles of components 

Quota Server's role:   



Update quota configurations (e.g: quota=1000) to local nodes. Sync node list to local nodes (Each node keep a copy) upon node insert/delete. Collect & Redistribute quota at some time interval to keep distributed quota balance.

Node's role:   

Update local quota count. Run house keeping job to monitor local quota. Send quota share request to next node (mono-directional). Response to share quota to preceding node upon receiving quota share request.

Characteristic & Benefits 

Characteristic: 







Distributed quota on nodes, instead of centralized & synced quota count. Make use of consistent hashing to form a virtual ordered chain of nodes. Balance node quota by mono-directional flow of quota through the node chain.

Benefits: 





No need to sync quota with Quota Server to confirm quota usage very often. Avoiding heavy loading on Quota Server. Each internodes-sync would just happen between two nodes. Minimize sync operation overhead. Lower complexity of sync quota logic between nodes. 

Sync is    

Mono-directional In consistent order Circular Eventually balanced.

Potential problems – Dead/blocking node Potential issues: • Application issue in responding quota share request which broke the propagation of quota balance flow. (e.g: Dead Node) •

Solution: If the node cannot get response from next node, then it try to request quota from the next of next node instead.

Potential problems – Quota unbalance due to propagation delay •

•

Quota request loop is not quick enough to balance the local quotas. Solution: –

–

–

We can tune propagation speed by tuning Quota monitoring house keeping job interval. Total time for a complete loop is around N*T, where N is number of nodes, and T is house keeping job interval. Say, 10 nodes with 5 second, a complete loop would take around 50 seconds. 5 seconds interval by default. 2 or lower value seconds interval for high usage systems.

Potential problems – False negative quota blocking in threshold cases •

•

Some false negative quota blocking issues may happen when it is very close to the quota threshold. It is because of used out of local quota while actually there are still valid quota in other nodes. Solution: –

When it reached some threshold situation, all nodes would turn into threshold mode •

•

•

Temp quota is added to all nodes to make sure that every node has sufficient quota to work out for accepting remaining number of quota usage. The tradeoff is that, in worst case there may be (N-1)*Tq excess quota, where N is number of nodes and Tq is maximum potential temp quota quantity. With reasonable configuration tuning, and the assumption that small amount of excess quota usage is tolerable, this can solve the false negative quota blocking issue.

Improvements 

Improvements 



The distributed quota may not ideally balanced due to propagation delay, even we set a very short house keeping job interval. Proposed improvement 

 



By using two hash methods, we can have the 2nd virtual chain instead of a single virtual chain of nodes. Hence, each node would have When each time node sending quota share request, it can choose to request which one of the two『next-node』 randomly. It would make the distributed quota much more balanced.

Distributed Node with Distributed Quota System (DNDQS).pdf ...

There was a problem previewing this document. Retrying... Download. Connect more apps... Try one of the apps below to open or edit this item. Distributed ...
Download PDF
459KB Sizes 4 Downloads 295 Views
Report

Recommend Documents

Conveyor carousel with distributed drive system
Nov 23, 2011 - poWer, loWer energy use, closed loop system monitoring and reduced ... This application is a reissue of US. patent application Ser. No. 12/128 ...
Distributed Operating System
IJRIT International Journal of Research in Information Technology, Volume 1, ... control unifies the different computers into a single integrated compute and ... resources, connections between these processes, and mappings of events ... of excellent
A distributed system architecture for a distributed ...
Advances in communications technology, development of powerful desktop workstations, and increased user demands for sophisticated applications are rapidly changing computing from a traditional centralized model to a distributed one. The tools and ser
Distributed File System
Hadoop file. System. Clustered- based, asymmetric. , parallel, object based. Statef ul ... File System http://hadoop.apache.org/core/docs/current/hdfs_de sign.html.
DISTRIBUTED PARAMETER ESTIMATION WITH SELECTIVE ...
shared with neighboring nodes, and a local consensus estimate is ob- tained by .... The complexity of the update depends on the update rule, f[·], employed at ...
Fault Tolerance in Distributed System - IJRIT
Fault Tolerance is an important issue in Distributed Computing. ... The partial failure is the key problem of the distributed system, .... architecture and design.
Deadlock Detection & Deadlock Prevention of Distributed System
deadlock detection and prevention approaches for distributed database. ... aspects of distributed systems, they are harder to detect, avoid, and prevent.
Deadlock in Distributed Operating System
Examples are given to illustrate these methods for avoiding file .... of files, such communications are never required to handle processes using only local files.
Fault Tolerance in Distributed System - IJRIT
Fault-tolerant describes a computer system or component designed so that, in the event that a component fails, a backup component or procedure can immediately ... millions of computing devices are working altogether and these millions of ...
DRMonitor – A Distributed Resource Monitoring System
classroom, involving 10 personal computers, is analyzed. Section 6 reviews related ... IP address: this is the node's Internet Protocol address. The IP address ...
Distributed File System Using Java
Though previous distributed file system such as Hadoop Distributed File System, ... able to quickly determine if potential users were able to understand the interface object. ... architecture being employed that is either client-server or peer-peer.
DISTRIBUTED ACOUSTIC MODELING WITH ... - Research at Google
best rescoring framework for Google Voice Search. 87,000 hours of training .... serving system (SSTable service) with S servers each holding. 1/S-th of the data.
Distributed Average Consensus With Dithered ... - IEEE Xplore
computation of averages of the node data over networks with band- width/power constraints or large volumes of data. Distributed averaging algorithms fail to ...
Distributed Averaging with Quantized Communication ...
Ji Liu2. Tamer Basar2. Behçet Açıkmese1. Abstract—Distributed algorithms are the key to enabling effective large scale distributed control systems, which present ..... xi(k)−⌊xi(k)⌋ be the decimal part of a node's local estimate value; the
DISTRIBUTED AVERAGE CONSENSUS WITH ...
“best constant” [1], is to set the neighboring edge weights to a constant ... The suboptimality of the best constant ... The degree of the node i is denoted di. |Ni|.
Distributed Programming with MapReduce
Jun 4, 2009 - a programming system for large-scale data processing ... save word_count to persistent storage … → will take .... locality. ○ backup tasks ...
Constructing Reliable Distributed Communication Systems with ...
liable distributed object computing systems with CORBA. First, we examine the .... backed by Lotus, Apple, IBM, Borland, MCI, Oracle, Word-. Perfect, and Novell) ...
Disorderly Distributed Programming with Bloom
Mutable for a short period. 2. Immutable forever after. • Example: bank accounts at end-of-day. • Example: distributed GC. – Once (global) refcount = 0, remains 0 ...
Distributed Verification and Hardness of Distributed ... - ETH TIK
and by the INRIA project GANG. Also supported by a France-Israel cooperation grant (“Mutli-Computing” project) from the France Ministry of Science and Israel ...
Distributed Verification and Hardness of Distributed ... - ETH TIK
C.2.4 [Computer Systems Organization]: Computer-. Communication Networks—Distributed Systems; F.0 [Theory of Computation]: General; G.2.2 [Mathematics ...
Distributed Creativity.pdf
the roles of creator and audience that underpins the. model of distributed creativity that we propose. Specifically, we believe that free association of ideas as.
Distributed Random Walks
Random walks play a central role in computer science, spanning a wide range of areas in ..... u uniformly at random and forwards C to u after incrementing the counter on the coupon to i. ...... IEEE Computer Society, Washington, DC, 218–223.
Efficient Distributed Quantum Computing
Nov 16, 2012 - tum circuit to a distributed quantum computer in which each ... Additionally, we prove that this is the best you can do; a 1D nearest neighbour machine .... Of course there is a price to pay: the overhead depends on the topology ...