The Need for Complex Analytics from Forwarding Pipelines

Editor's Notes

• #2: Separate actual measurement hardware logic from its control and the analysis logic to enable high dynamic on demand probing interface Define a simple abstraction that allows the control logic to program and specify what to measure and where to send the data  Measurement controller collects network-wide data and associates it with the network graph Measurement controller can store the data in memory for real time analytics or dump it in a database for off-line analytics   Measurement controller provides abstractions and APIs to make it easier to write real time and non real time analytics apps  Utilize NFVs as analytics functions for analytics to be in the data plane Think Pure…Abstract Pure… Architect for Abstractions and extensibility of Probes and Sensors Implement open Interfaces and Modularity to enable dynamic Probe instruction Sets Extend P4 to Complex Analytics without compromising purity
• #3: Virtual Probe Observability and Analytics, utilizing a common compilation and abstraction model (Unified Probe Analytic Orchestration) On Demand Flow Monitory/Filtering and QoE measurement DDOS attach Detection Elastic Traffic Management Classification & control
• #4: SDN & NFV have not played the appropriate role on deep observability
• #5: SDN/NFV been focused on virtualizing many network and IT business functions. The emerging platforms consolidates many proprietary network environments into a Open platform based on commodity HW/SW for increased dev-op models
• #7: Observability Abstraction Resource state dissemination/collection Ability to collect state of resources/elements Ability to observe probes adaptively and on Demand Programming abstraction Ability to program traffic forwarding rules/policies Ability to program the Probes Control blocks Configuration abstraction Ability to configure the resources bases on policies Ability to configure the appropriate Probes for real time needs
• #11: Modern Analytics require a dynamic and programmable underlay of capturing data (probes) High level languages such as P4 and Match/Action structures allow for the creation of such rule based environments Doing this using CPUs consumes too much of the CPU limiting the number of VMs that can be deployed per server (for Applications or VNFs) This meant that network analytics were limited by CPU processing at the edge and too much bandwidth at the switches. SmartNICs can aid this process greatly: More programmable, more flexible and more performant SmartNICs include: Fully programmble solutions Netronome Agilio 10, 25, 40, 100 Gbps solutions, Cavium LiquidIO and configurable solutions from Intel and others This opportunity allows the definition of new dynamic analytics within the SmartNIC to improve network monitoring FIXED FUNCTION CHARACTERISTICS Functionality “baked-in” to silicon. Good if it implements exactly what you need. Bad if it doesn’t. Some Pipelines are more configurable but not fully flexible PROGRAMMABLE CHARACTERISTICS Packets processed by “sea of workers”- threaded architectures with multi/many core configuration Optimized memory architecture for the application High-bandwidth on-chip switch-fabric links memories and workers Many core solutions such as Netronome can scale to very high processing levels
• #12: Allows lightweight, fast monitoring by utilising the Netronome Flow Cache’s flexibility to store observations Flow cache is programmable and flexible, stores a minimum of 250K flows with 1KB of stats each using built in 2GB of DRAM (Agilio-CX) The vProbe app must interpret the flow stats and features, react and send aggregated information to rack level controllers
• #13: Fast aggregation, classification and filtering in under 2s required to achieve aim of <15s analytics