SlideShare a Scribd company logo
Improving Robustness in Distributed Systems  Per Bergqvist [email_address] Erlang User Conference 2001 (courtesy CellPoint Systems AB)
Design base Cluster of cooperating hosts Erlang and C COTS hardware based Unix based (i.e. Solaris or Linux) 10/100/1000 base-T back plane (”system area network”)
Cluster Shared, distributed, system configuration Each host have ONE cluster controller Dispatch and supervise worker tasks Master cluster controller: holds configuration database (persistent replica) Slave cluster controller: gets configuration from master cluster controllers Cluster is DOWN when all master cluster controllers are inaccessible
Typical system Firewall Switch Traffic Control
Cluster Key Benefits Single system view Enforces decoupling of parts of O&M from actual traffic processing
Implementing a cluster Cluster->Host->Node->NodeData  Cluster global parameters Subscription mechanisms for conf. changes Mnesia as configuration database on master cluster controllers Homebrewn configuration distribution to slave controllers (NOT using mnesia) (Worker) node supervision
Mnesia gotchas First distributed node startup  Disallow writes when all replicas not accessible Use timeout on table load and force load
... BUT ... TCP based distribution Network partitioning
Network parameters Align TCP retransmission intervals w/ Erlang heartbeats Align TCP and IP rerouting parameters
Typical system II:  Dual back plane Firewall Switch Traffic Control
Erlang multi-homing problem Host A Host B Host C
Multi-home Erlang w/ TCP Add an alias interface to loop back i/f Patch tcp distribution to bind to alias Publish alias interface on (all wanted) via real hw i/f’s Method 1: Static routes and gratuitous/proxy arp Method 2: Use new (routing) protocol
ARP method Implement a utility to: - broadcast unsolicited ARP responses - respond to ARP requests  for the alias i/f address Add static routes on all far end systems NOTE: all real i/f needs to be on same IP subnet
New routing protocol Broadcast (Ethernet frames) what you have, including interface priority Let the far end select path based on  what/when they receive Far end dynamically sets up host routes Use short retransmission intervals
Erlang multi-homing resolved ? Host A Host B Host C
Summing up Erlang can support multihoming with some additional work By using loop back alias i/f, link failure becomes a routing problem (peer-peer association is kept intact) Solaris TCP/IP stack parameters are: - hard to find (only in out-of-date app. notes) - hard to set ”right” - host global A distribution mechanism with built-in support for multi-homing preferred
Erlang Distribution over SCTP Per Bergqvist et al [email_address] Erlang User Conference 2002
Ad

More Related Content

What's hot (20)

Basics of sockets
Basics of socketsBasics of sockets
Basics of sockets
AviNash ChaVhan
 
Lect16
Lect16Lect16
Lect16
Vandana Gupta
 
Lab 4 final report
Lab 4 final reportLab 4 final report
Lab 4 final report
Kyle Villano
 
Networking in python by Rj
Networking in python by RjNetworking in python by Rj
Networking in python by Rj
Shree M.L.Kakadiya MCA mahila college, Amreli
 
What is Socket Programming in Python | Edureka
What is Socket Programming in Python | EdurekaWhat is Socket Programming in Python | Edureka
What is Socket Programming in Python | Edureka
Edureka!
 
Socket programming-in-python
Socket programming-in-pythonSocket programming-in-python
Socket programming-in-python
Yuvaraja Ravi
 
Building the Internet of Things with Thingsquare and Contiki - day 2 part 1
Building the Internet of Things with Thingsquare and Contiki - day 2 part 1Building the Internet of Things with Thingsquare and Contiki - day 2 part 1
Building the Internet of Things with Thingsquare and Contiki - day 2 part 1
Adam Dunkels
 
Messaging With Erlang And Jabber
Messaging With  Erlang And  JabberMessaging With  Erlang And  Jabber
Messaging With Erlang And Jabber
l xf
 
Python Sockets
Python SocketsPython Sockets
Python Sockets
pythontic
 
Sockets in unix
Sockets in unixSockets in unix
Sockets in unix
swtjerin4u
 
An Introduction to OMNeT++ 5.1
An Introduction to OMNeT++ 5.1An Introduction to OMNeT++ 5.1
An Introduction to OMNeT++ 5.1
Alpen-Adria-Universität
 
Arp and rarp
Arp and rarpArp and rarp
Arp and rarp
Mohd Arif
 
Sockets and Socket-Buffer
Sockets and Socket-BufferSockets and Socket-Buffer
Sockets and Socket-Buffer
Sourav Punoriyar
 
Programming using Open Mp
Programming using Open MpProgramming using Open Mp
Programming using Open Mp
Anshul Sharma
 
Introduction to tcp ip linux networking
Introduction to tcp ip   linux networkingIntroduction to tcp ip   linux networking
Introduction to tcp ip linux networking
Sreenatha Reddy K R
 
Linker and loader upload
Linker and loader   uploadLinker and loader   upload
Linker and loader upload
Bin Yang
 
DUSK - Develop at Userland Install into Kernel
DUSK - Develop at Userland Install into KernelDUSK - Develop at Userland Install into Kernel
DUSK - Develop at Userland Install into Kernel
Alexey Smirnov
 
hajer
hajerhajer
hajer
ra na
 
Socket programming
Socket programmingSocket programming
Socket programming
chandramouligunnemeda
 
Arp and rarp
Arp and rarpArp and rarp
Arp and rarp
राहुल खेडेकर
 
Lab 4 final report
Lab 4 final reportLab 4 final report
Lab 4 final report
Kyle Villano
 
What is Socket Programming in Python | Edureka
What is Socket Programming in Python | EdurekaWhat is Socket Programming in Python | Edureka
What is Socket Programming in Python | Edureka
Edureka!
 
Socket programming-in-python
Socket programming-in-pythonSocket programming-in-python
Socket programming-in-python
Yuvaraja Ravi
 
Building the Internet of Things with Thingsquare and Contiki - day 2 part 1
Building the Internet of Things with Thingsquare and Contiki - day 2 part 1Building the Internet of Things with Thingsquare and Contiki - day 2 part 1
Building the Internet of Things with Thingsquare and Contiki - day 2 part 1
Adam Dunkels
 
Messaging With Erlang And Jabber
Messaging With  Erlang And  JabberMessaging With  Erlang And  Jabber
Messaging With Erlang And Jabber
l xf
 
Python Sockets
Python SocketsPython Sockets
Python Sockets
pythontic
 
Sockets in unix
Sockets in unixSockets in unix
Sockets in unix
swtjerin4u
 
Arp and rarp
Arp and rarpArp and rarp
Arp and rarp
Mohd Arif
 
Programming using Open Mp
Programming using Open MpProgramming using Open Mp
Programming using Open Mp
Anshul Sharma
 
Introduction to tcp ip linux networking
Introduction to tcp ip   linux networkingIntroduction to tcp ip   linux networking
Introduction to tcp ip linux networking
Sreenatha Reddy K R
 
Linker and loader upload
Linker and loader   uploadLinker and loader   upload
Linker and loader upload
Bin Yang
 
DUSK - Develop at Userland Install into Kernel
DUSK - Develop at Userland Install into KernelDUSK - Develop at Userland Install into Kernel
DUSK - Develop at Userland Install into Kernel
Alexey Smirnov
 
hajer
hajerhajer
hajer
ra na
 

Viewers also liked (20)

A Virtual World Distributed Server Developed In Erlang As A Tool For Analysin...
A Virtual World Distributed Server Developed In Erlang As A Tool For Analysin...A Virtual World Distributed Server Developed In Erlang As A Tool For Analysin...
A Virtual World Distributed Server Developed In Erlang As A Tool For Analysin...
l xf
 
QoS of multiplayer games over 802.11
QoS of multiplayer games over 802.11QoS of multiplayer games over 802.11
QoS of multiplayer games over 802.11
muraliashwin
 
What is Eddystone
What is EddystoneWhat is Eddystone
What is Eddystone
bfonics
 
Milieu Quiz 2015: Quizzical (Short Visual Connect 2)
Milieu Quiz 2015: Quizzical (Short Visual Connect 2)Milieu Quiz 2015: Quizzical (Short Visual Connect 2)
Milieu Quiz 2015: Quizzical (Short Visual Connect 2)
pcsagk
 
Online Quiz Daivik
Online Quiz DaivikOnline Quiz Daivik
Online Quiz Daivik
Daivik De
 
Android Build Server
Android Build ServerAndroid Build Server
Android Build Server
Trey Robinson
 
Make your-game-multiplayer
Make your-game-multiplayerMake your-game-multiplayer
Make your-game-multiplayer
Andrew Lee
 
Building Multiplayer Games (w/ Unity)
Building Multiplayer Games (w/ Unity)Building Multiplayer Games (w/ Unity)
Building Multiplayer Games (w/ Unity)
Noam Gat
 
Server side game_development
Server side game_developmentServer side game_development
Server side game_development
Yekmer Simsek
 
Intro to Massively Multiplayer Online Game (MMOG) Design
Intro to Massively Multiplayer Online Game (MMOG) DesignIntro to Massively Multiplayer Online Game (MMOG) Design
Intro to Massively Multiplayer Online Game (MMOG) Design
Christopher Mohritz
 
Cloud Gaming - A Green Solution to Massive Multiplayer Online Games
Cloud Gaming - A Green Solution to Massive Multiplayer Online Games Cloud Gaming - A Green Solution to Massive Multiplayer Online Games
Cloud Gaming - A Green Solution to Massive Multiplayer Online Games
Suhas Urs
 
Designing Products for Cellular Assembly
Designing Products for Cellular AssemblyDesigning Products for Cellular Assembly
Designing Products for Cellular Assembly
Advent Design Corporation
 
Building Real-Time Applications with Android and WebSockets
Building Real-Time Applications with Android and WebSocketsBuilding Real-Time Applications with Android and WebSockets
Building Real-Time Applications with Android and WebSockets
Sergi Almar i Graupera
 
Quiz For Money
Quiz For MoneyQuiz For Money
Quiz For Money
seo.presentation
 
Introduction to android
Introduction to androidIntroduction to android
Introduction to android
zeelpatel0504
 
Php code for online quiz
Php code for online quizPhp code for online quiz
Php code for online quiz
hnyb1002
 
Android tutorial ppt
Android tutorial pptAndroid tutorial ppt
Android tutorial ppt
Rehna Renu
 
Android Fast Track CRUD Android PHP MySql
Android Fast Track CRUD Android PHP MySqlAndroid Fast Track CRUD Android PHP MySql
Android Fast Track CRUD Android PHP MySql
Agus Haryanto
 
Ent quiz questions
Ent quiz questionsEnt quiz questions
Ent quiz questions
Sourjya Chowdhury
 
Basic android-ppt
Basic android-pptBasic android-ppt
Basic android-ppt
Srijib Roy
 
A Virtual World Distributed Server Developed In Erlang As A Tool For Analysin...
A Virtual World Distributed Server Developed In Erlang As A Tool For Analysin...A Virtual World Distributed Server Developed In Erlang As A Tool For Analysin...
A Virtual World Distributed Server Developed In Erlang As A Tool For Analysin...
l xf
 
QoS of multiplayer games over 802.11
QoS of multiplayer games over 802.11QoS of multiplayer games over 802.11
QoS of multiplayer games over 802.11
muraliashwin
 
What is Eddystone
What is EddystoneWhat is Eddystone
What is Eddystone
bfonics
 
Milieu Quiz 2015: Quizzical (Short Visual Connect 2)
Milieu Quiz 2015: Quizzical (Short Visual Connect 2)Milieu Quiz 2015: Quizzical (Short Visual Connect 2)
Milieu Quiz 2015: Quizzical (Short Visual Connect 2)
pcsagk
 
Online Quiz Daivik
Online Quiz DaivikOnline Quiz Daivik
Online Quiz Daivik
Daivik De
 
Android Build Server
Android Build ServerAndroid Build Server
Android Build Server
Trey Robinson
 
Make your-game-multiplayer
Make your-game-multiplayerMake your-game-multiplayer
Make your-game-multiplayer
Andrew Lee
 
Building Multiplayer Games (w/ Unity)
Building Multiplayer Games (w/ Unity)Building Multiplayer Games (w/ Unity)
Building Multiplayer Games (w/ Unity)
Noam Gat
 
Server side game_development
Server side game_developmentServer side game_development
Server side game_development
Yekmer Simsek
 
Intro to Massively Multiplayer Online Game (MMOG) Design
Intro to Massively Multiplayer Online Game (MMOG) DesignIntro to Massively Multiplayer Online Game (MMOG) Design
Intro to Massively Multiplayer Online Game (MMOG) Design
Christopher Mohritz
 
Cloud Gaming - A Green Solution to Massive Multiplayer Online Games
Cloud Gaming - A Green Solution to Massive Multiplayer Online Games Cloud Gaming - A Green Solution to Massive Multiplayer Online Games
Cloud Gaming - A Green Solution to Massive Multiplayer Online Games
Suhas Urs
 
Building Real-Time Applications with Android and WebSockets
Building Real-Time Applications with Android and WebSocketsBuilding Real-Time Applications with Android and WebSockets
Building Real-Time Applications with Android and WebSockets
Sergi Almar i Graupera
 
Introduction to android
Introduction to androidIntroduction to android
Introduction to android
zeelpatel0504
 
Php code for online quiz
Php code for online quizPhp code for online quiz
Php code for online quiz
hnyb1002
 
Android tutorial ppt
Android tutorial pptAndroid tutorial ppt
Android tutorial ppt
Rehna Renu
 
Android Fast Track CRUD Android PHP MySql
Android Fast Track CRUD Android PHP MySqlAndroid Fast Track CRUD Android PHP MySql
Android Fast Track CRUD Android PHP MySql
Agus Haryanto
 
Basic android-ppt
Basic android-pptBasic android-ppt
Basic android-ppt
Srijib Roy
 
Ad

Similar to Improving Robustness In Distributed Systems (20)

Ccna Imp Guide
Ccna Imp GuideCcna Imp Guide
Ccna Imp Guide
abhijitgnbbl
 
Next-Generation Best Practices for VMware and Storage
Next-Generation Best Practices for VMware and StorageNext-Generation Best Practices for VMware and Storage
Next-Generation Best Practices for VMware and Storage
Scott Lowe
 
introduction to linux kernel tcp/ip ptocotol stack
introduction to linux kernel tcp/ip ptocotol stack introduction to linux kernel tcp/ip ptocotol stack
introduction to linux kernel tcp/ip ptocotol stack
monad bobo
 
10 high speedla-ns
10 high speedla-ns10 high speedla-ns
10 high speedla-ns
Hattori Sidek
 
Solaris multipathing
Solaris multipathingSolaris multipathing
Solaris multipathing
Bui Van Cuong
 
Training Day Slides
Training Day SlidesTraining Day Slides
Training Day Slides
adam_merritt
 
Networking in Gnu/Linux
Networking in Gnu/LinuxNetworking in Gnu/Linux
Networking in Gnu/Linux
Ahmed Mekkawy
 
link layer
link layerlink layer
link layer
GS Kosta
 
Juniper Trouble Shooting
Juniper Trouble ShootingJuniper Trouble Shooting
Juniper Trouble Shooting
Mike(Haobin) Zheng
 
Network Topologies
Network TopologiesNetwork Topologies
Network Topologies
Jason Hando
 
linux networking laboratory presentation .pptx
linux networking laboratory presentation .pptxlinux networking laboratory presentation .pptx
linux networking laboratory presentation .pptx
AnuradhaJadiya1
 
He Pi Xii2003
He Pi Xii2003He Pi Xii2003
He Pi Xii2003
FNian
 
Lect2_MS_Networks OSI ,TCP model,ETHERNET.pptx
Lect2_MS_Networks OSI ,TCP model,ETHERNET.pptxLect2_MS_Networks OSI ,TCP model,ETHERNET.pptx
Lect2_MS_Networks OSI ,TCP model,ETHERNET.pptx
MuhammadHamza117822
 
Slides for Week 5 - Lec 1
Slides for Week 5 - Lec 1Slides for Week 5 - Lec 1
Slides for Week 5 - Lec 1
webhostingguy
 
2011 q1-indy-vmug
2011 q1-indy-vmug2011 q1-indy-vmug
2011 q1-indy-vmug
Adam Eckerle
 
Ethernet VPN (EVPN) EVerything Provider Needs
Ethernet VPN (EVPN) EVerything Provider NeedsEthernet VPN (EVPN) EVerything Provider Needs
Ethernet VPN (EVPN) EVerything Provider Needs
CSUC - Consorci de Serveis Universitaris de Catalunya
 
Network and distributed systems
Network and distributed systemsNetwork and distributed systems
Network and distributed systems
Sri Prasanna
 
Basic Introduction to Technology (networking).pdf
Basic Introduction to Technology (networking).pdfBasic Introduction to Technology (networking).pdf
Basic Introduction to Technology (networking).pdf
tthind
 
Ambari metrics system - Apache ambari meetup (DataWorks Summit 2017)
Ambari metrics system - Apache ambari meetup (DataWorks Summit 2017)Ambari metrics system - Apache ambari meetup (DataWorks Summit 2017)
Ambari metrics system - Apache ambari meetup (DataWorks Summit 2017)
Aravindan Vijayan
 
Intro (Distributed computing)
Intro (Distributed computing)Intro (Distributed computing)
Intro (Distributed computing)
Sri Prasanna
 
Next-Generation Best Practices for VMware and Storage
Next-Generation Best Practices for VMware and StorageNext-Generation Best Practices for VMware and Storage
Next-Generation Best Practices for VMware and Storage
Scott Lowe
 
introduction to linux kernel tcp/ip ptocotol stack
introduction to linux kernel tcp/ip ptocotol stack introduction to linux kernel tcp/ip ptocotol stack
introduction to linux kernel tcp/ip ptocotol stack
monad bobo
 
Solaris multipathing
Solaris multipathingSolaris multipathing
Solaris multipathing
Bui Van Cuong
 
Training Day Slides
Training Day SlidesTraining Day Slides
Training Day Slides
adam_merritt
 
Networking in Gnu/Linux
Networking in Gnu/LinuxNetworking in Gnu/Linux
Networking in Gnu/Linux
Ahmed Mekkawy
 
link layer
link layerlink layer
link layer
GS Kosta
 
Network Topologies
Network TopologiesNetwork Topologies
Network Topologies
Jason Hando
 
linux networking laboratory presentation .pptx
linux networking laboratory presentation .pptxlinux networking laboratory presentation .pptx
linux networking laboratory presentation .pptx
AnuradhaJadiya1
 
He Pi Xii2003
He Pi Xii2003He Pi Xii2003
He Pi Xii2003
FNian
 
Lect2_MS_Networks OSI ,TCP model,ETHERNET.pptx
Lect2_MS_Networks OSI ,TCP model,ETHERNET.pptxLect2_MS_Networks OSI ,TCP model,ETHERNET.pptx
Lect2_MS_Networks OSI ,TCP model,ETHERNET.pptx
MuhammadHamza117822
 
Slides for Week 5 - Lec 1
Slides for Week 5 - Lec 1Slides for Week 5 - Lec 1
Slides for Week 5 - Lec 1
webhostingguy
 
Network and distributed systems
Network and distributed systemsNetwork and distributed systems
Network and distributed systems
Sri Prasanna
 
Basic Introduction to Technology (networking).pdf
Basic Introduction to Technology (networking).pdfBasic Introduction to Technology (networking).pdf
Basic Introduction to Technology (networking).pdf
tthind
 
Ambari metrics system - Apache ambari meetup (DataWorks Summit 2017)
Ambari metrics system - Apache ambari meetup (DataWorks Summit 2017)Ambari metrics system - Apache ambari meetup (DataWorks Summit 2017)
Ambari metrics system - Apache ambari meetup (DataWorks Summit 2017)
Aravindan Vijayan
 
Intro (Distributed computing)
Intro (Distributed computing)Intro (Distributed computing)
Intro (Distributed computing)
Sri Prasanna
 
Ad

More from l xf (6)

Asynchronous Io Programming
Asynchronous Io ProgrammingAsynchronous Io Programming
Asynchronous Io Programming
l xf
 
The Proactor Pattern
The Proactor PatternThe Proactor Pattern
The Proactor Pattern
l xf
 
Scalable Networking
Scalable NetworkingScalable Networking
Scalable Networking
l xf
 
The Migration From Erlang To Otp A Case Study Of A Heavy Duty Tcpip Clients...
The Migration From Erlang To Otp   A Case Study Of A Heavy Duty Tcpip Clients...The Migration From Erlang To Otp   A Case Study Of A Heavy Duty Tcpip Clients...
The Migration From Erlang To Otp A Case Study Of A Heavy Duty Tcpip Clients...
l xf
 
Concurrency And Erlang
Concurrency And ErlangConcurrency And Erlang
Concurrency And Erlang
l xf
 
Learning Erlang And Developing A Sip Server Stack With 30k Potential Users
Learning Erlang And Developing A Sip Server Stack With 30k Potential UsersLearning Erlang And Developing A Sip Server Stack With 30k Potential Users
Learning Erlang And Developing A Sip Server Stack With 30k Potential Users
l xf
 
Asynchronous Io Programming
Asynchronous Io ProgrammingAsynchronous Io Programming
Asynchronous Io Programming
l xf
 
The Proactor Pattern
The Proactor PatternThe Proactor Pattern
The Proactor Pattern
l xf
 
Scalable Networking
Scalable NetworkingScalable Networking
Scalable Networking
l xf
 
The Migration From Erlang To Otp A Case Study Of A Heavy Duty Tcpip Clients...
The Migration From Erlang To Otp   A Case Study Of A Heavy Duty Tcpip Clients...The Migration From Erlang To Otp   A Case Study Of A Heavy Duty Tcpip Clients...
The Migration From Erlang To Otp A Case Study Of A Heavy Duty Tcpip Clients...
l xf
 
Concurrency And Erlang
Concurrency And ErlangConcurrency And Erlang
Concurrency And Erlang
l xf
 
Learning Erlang And Developing A Sip Server Stack With 30k Potential Users
Learning Erlang And Developing A Sip Server Stack With 30k Potential UsersLearning Erlang And Developing A Sip Server Stack With 30k Potential Users
Learning Erlang And Developing A Sip Server Stack With 30k Potential Users
l xf
 

Recently uploaded (20)

Design pattern talk by Kaya Weers - 2025 (v2)
Design pattern talk by Kaya Weers - 2025 (v2)Design pattern talk by Kaya Weers - 2025 (v2)
Design pattern talk by Kaya Weers - 2025 (v2)
Kaya Weers
 
論文紹介:"InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning" ...
論文紹介:"InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning" ...論文紹介:"InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning" ...
論文紹介:"InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning" ...
Toru Tamaki
 
May Patch Tuesday
May Patch TuesdayMay Patch Tuesday
May Patch Tuesday
Ivanti
 
Agentic Automation - Delhi UiPath Community Meetup
Agentic Automation - Delhi UiPath Community MeetupAgentic Automation - Delhi UiPath Community Meetup
Agentic Automation - Delhi UiPath Community Meetup
Manoj Batra (1600 + Connections)
 
Crazy Incentives and How They Kill Security. How Do You Turn the Wheel?
Crazy Incentives and How They Kill Security. How Do You Turn the Wheel?Crazy Incentives and How They Kill Security. How Do You Turn the Wheel?
Crazy Incentives and How They Kill Security. How Do You Turn the Wheel?
Christian Folini
 
Artificial_Intelligence_in_Everyday_Life.pptx
Artificial_Intelligence_in_Everyday_Life.pptxArtificial_Intelligence_in_Everyday_Life.pptx
Artificial_Intelligence_in_Everyday_Life.pptx
03ANMOLCHAURASIYA
 
Cybersecurity Tools and Technologies - Microsoft Certificate
Cybersecurity Tools and Technologies - Microsoft CertificateCybersecurity Tools and Technologies - Microsoft Certificate
Cybersecurity Tools and Technologies - Microsoft Certificate
VICTOR MAESTRE RAMIREZ
 
Build With AI - In Person Session Slides.pdf
Build With AI - In Person Session Slides.pdfBuild With AI - In Person Session Slides.pdf
Build With AI - In Person Session Slides.pdf
Google Developer Group - Harare
 
Limecraft Webinar - 2025.3 release, featuring Content Delivery, Graphic Conte...
Limecraft Webinar - 2025.3 release, featuring Content Delivery, Graphic Conte...Limecraft Webinar - 2025.3 release, featuring Content Delivery, Graphic Conte...
Limecraft Webinar - 2025.3 release, featuring Content Delivery, Graphic Conte...
Maarten Verwaest
 
DNF 2.0 Implementations Challenges in Nepal
DNF 2.0 Implementations Challenges in NepalDNF 2.0 Implementations Challenges in Nepal
DNF 2.0 Implementations Challenges in Nepal
ICT Frame Magazine Pvt. Ltd.
 
Config 2025 presentation recap covering both days
Config 2025 presentation recap covering both daysConfig 2025 presentation recap covering both days
Config 2025 presentation recap covering both days
TrishAntoni1
 
Kit-Works Team Study_팀스터디_김한솔_nuqs_20250509.pdf
Kit-Works Team Study_팀스터디_김한솔_nuqs_20250509.pdfKit-Works Team Study_팀스터디_김한솔_nuqs_20250509.pdf
Kit-Works Team Study_팀스터디_김한솔_nuqs_20250509.pdf
Wonjun Hwang
 
An Overview of Salesforce Health Cloud & How is it Transforming Patient Care
An Overview of Salesforce Health Cloud & How is it Transforming Patient CareAn Overview of Salesforce Health Cloud & How is it Transforming Patient Care
An Overview of Salesforce Health Cloud & How is it Transforming Patient Care
Cyntexa
 
fennec fox optimization algorithm for optimal solution
fennec fox optimization algorithm for optimal solutionfennec fox optimization algorithm for optimal solution
fennec fox optimization algorithm for optimal solution
shallal2
 
Who's choice? Making decisions with and about Artificial Intelligence, Keele ...
Who's choice? Making decisions with and about Artificial Intelligence, Keele ...Who's choice? Making decisions with and about Artificial Intelligence, Keele ...
Who's choice? Making decisions with and about Artificial Intelligence, Keele ...
Alan Dix
 
React Native for Business Solutions: Building Scalable Apps for Success
React Native for Business Solutions: Building Scalable Apps for SuccessReact Native for Business Solutions: Building Scalable Apps for Success
React Native for Business Solutions: Building Scalable Apps for Success
Amelia Swank
 
ACE Aarhus - Team'25 wrap-up presentation
ACE Aarhus - Team'25 wrap-up presentationACE Aarhus - Team'25 wrap-up presentation
ACE Aarhus - Team'25 wrap-up presentation
DanielEriksen5
 
Unlocking Generative AI in your Web Apps
Unlocking Generative AI in your Web AppsUnlocking Generative AI in your Web Apps
Unlocking Generative AI in your Web Apps
Maximiliano Firtman
 
On-Device or Remote? On the Energy Efficiency of Fetching LLM-Generated Conte...
On-Device or Remote? On the Energy Efficiency of Fetching LLM-Generated Conte...On-Device or Remote? On the Energy Efficiency of Fetching LLM-Generated Conte...
On-Device or Remote? On the Energy Efficiency of Fetching LLM-Generated Conte...
Ivano Malavolta
 
Building the Customer Identity Community, Together.pdf
Building the Customer Identity Community, Together.pdfBuilding the Customer Identity Community, Together.pdf
Building the Customer Identity Community, Together.pdf
Cheryl Hung
 
Design pattern talk by Kaya Weers - 2025 (v2)
Design pattern talk by Kaya Weers - 2025 (v2)Design pattern talk by Kaya Weers - 2025 (v2)
Design pattern talk by Kaya Weers - 2025 (v2)
Kaya Weers
 
論文紹介:"InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning" ...
論文紹介:"InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning" ...論文紹介:"InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning" ...
論文紹介:"InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning" ...
Toru Tamaki
 
May Patch Tuesday
May Patch TuesdayMay Patch Tuesday
May Patch Tuesday
Ivanti
 
Crazy Incentives and How They Kill Security. How Do You Turn the Wheel?
Crazy Incentives and How They Kill Security. How Do You Turn the Wheel?Crazy Incentives and How They Kill Security. How Do You Turn the Wheel?
Crazy Incentives and How They Kill Security. How Do You Turn the Wheel?
Christian Folini
 
Artificial_Intelligence_in_Everyday_Life.pptx
Artificial_Intelligence_in_Everyday_Life.pptxArtificial_Intelligence_in_Everyday_Life.pptx
Artificial_Intelligence_in_Everyday_Life.pptx
03ANMOLCHAURASIYA
 
Cybersecurity Tools and Technologies - Microsoft Certificate
Cybersecurity Tools and Technologies - Microsoft CertificateCybersecurity Tools and Technologies - Microsoft Certificate
Cybersecurity Tools and Technologies - Microsoft Certificate
VICTOR MAESTRE RAMIREZ
 
Limecraft Webinar - 2025.3 release, featuring Content Delivery, Graphic Conte...
Limecraft Webinar - 2025.3 release, featuring Content Delivery, Graphic Conte...Limecraft Webinar - 2025.3 release, featuring Content Delivery, Graphic Conte...
Limecraft Webinar - 2025.3 release, featuring Content Delivery, Graphic Conte...
Maarten Verwaest
 
Config 2025 presentation recap covering both days
Config 2025 presentation recap covering both daysConfig 2025 presentation recap covering both days
Config 2025 presentation recap covering both days
TrishAntoni1
 
Kit-Works Team Study_팀스터디_김한솔_nuqs_20250509.pdf
Kit-Works Team Study_팀스터디_김한솔_nuqs_20250509.pdfKit-Works Team Study_팀스터디_김한솔_nuqs_20250509.pdf
Kit-Works Team Study_팀스터디_김한솔_nuqs_20250509.pdf
Wonjun Hwang
 
An Overview of Salesforce Health Cloud & How is it Transforming Patient Care
An Overview of Salesforce Health Cloud & How is it Transforming Patient CareAn Overview of Salesforce Health Cloud & How is it Transforming Patient Care
An Overview of Salesforce Health Cloud & How is it Transforming Patient Care
Cyntexa
 
fennec fox optimization algorithm for optimal solution
fennec fox optimization algorithm for optimal solutionfennec fox optimization algorithm for optimal solution
fennec fox optimization algorithm for optimal solution
shallal2
 
Who's choice? Making decisions with and about Artificial Intelligence, Keele ...
Who's choice? Making decisions with and about Artificial Intelligence, Keele ...Who's choice? Making decisions with and about Artificial Intelligence, Keele ...
Who's choice? Making decisions with and about Artificial Intelligence, Keele ...
Alan Dix
 
React Native for Business Solutions: Building Scalable Apps for Success
React Native for Business Solutions: Building Scalable Apps for SuccessReact Native for Business Solutions: Building Scalable Apps for Success
React Native for Business Solutions: Building Scalable Apps for Success
Amelia Swank
 
ACE Aarhus - Team'25 wrap-up presentation
ACE Aarhus - Team'25 wrap-up presentationACE Aarhus - Team'25 wrap-up presentation
ACE Aarhus - Team'25 wrap-up presentation
DanielEriksen5
 
Unlocking Generative AI in your Web Apps
Unlocking Generative AI in your Web AppsUnlocking Generative AI in your Web Apps
Unlocking Generative AI in your Web Apps
Maximiliano Firtman
 
On-Device or Remote? On the Energy Efficiency of Fetching LLM-Generated Conte...
On-Device or Remote? On the Energy Efficiency of Fetching LLM-Generated Conte...On-Device or Remote? On the Energy Efficiency of Fetching LLM-Generated Conte...
On-Device or Remote? On the Energy Efficiency of Fetching LLM-Generated Conte...
Ivano Malavolta
 
Building the Customer Identity Community, Together.pdf
Building the Customer Identity Community, Together.pdfBuilding the Customer Identity Community, Together.pdf
Building the Customer Identity Community, Together.pdf
Cheryl Hung
 

Improving Robustness In Distributed Systems

  • 1. Improving Robustness in Distributed Systems Per Bergqvist [email_address] Erlang User Conference 2001 (courtesy CellPoint Systems AB)
  • 2. Design base Cluster of cooperating hosts Erlang and C COTS hardware based Unix based (i.e. Solaris or Linux) 10/100/1000 base-T back plane (”system area network”)
  • 3. Cluster Shared, distributed, system configuration Each host have ONE cluster controller Dispatch and supervise worker tasks Master cluster controller: holds configuration database (persistent replica) Slave cluster controller: gets configuration from master cluster controllers Cluster is DOWN when all master cluster controllers are inaccessible
  • 4. Typical system Firewall Switch Traffic Control
  • 5. Cluster Key Benefits Single system view Enforces decoupling of parts of O&M from actual traffic processing
  • 6. Implementing a cluster Cluster->Host->Node->NodeData Cluster global parameters Subscription mechanisms for conf. changes Mnesia as configuration database on master cluster controllers Homebrewn configuration distribution to slave controllers (NOT using mnesia) (Worker) node supervision
  • 7. Mnesia gotchas First distributed node startup Disallow writes when all replicas not accessible Use timeout on table load and force load
  • 8. ... BUT ... TCP based distribution Network partitioning
  • 9. Network parameters Align TCP retransmission intervals w/ Erlang heartbeats Align TCP and IP rerouting parameters
  • 10. Typical system II: Dual back plane Firewall Switch Traffic Control
  • 11. Erlang multi-homing problem Host A Host B Host C
  • 12. Multi-home Erlang w/ TCP Add an alias interface to loop back i/f Patch tcp distribution to bind to alias Publish alias interface on (all wanted) via real hw i/f’s Method 1: Static routes and gratuitous/proxy arp Method 2: Use new (routing) protocol
  • 13. ARP method Implement a utility to: - broadcast unsolicited ARP responses - respond to ARP requests for the alias i/f address Add static routes on all far end systems NOTE: all real i/f needs to be on same IP subnet
  • 14. New routing protocol Broadcast (Ethernet frames) what you have, including interface priority Let the far end select path based on what/when they receive Far end dynamically sets up host routes Use short retransmission intervals
  • 15. Erlang multi-homing resolved ? Host A Host B Host C
  • 16. Summing up Erlang can support multihoming with some additional work By using loop back alias i/f, link failure becomes a routing problem (peer-peer association is kept intact) Solaris TCP/IP stack parameters are: - hard to find (only in out-of-date app. notes) - hard to set ”right” - host global A distribution mechanism with built-in support for multi-homing preferred
  • 17. Erlang Distribution over SCTP Per Bergqvist et al [email_address] Erlang User Conference 2002
  翻译: