Pipeline oriented programming

Pipeline Oriented
Programming
(DotNext 2021)
@ScottWlaschin
fsharpforfunandprofit.com/pipeline
Code examples will be in C# and F#

What is
“Pipeline-oriented Programming"?
Part I

Object-oriented
Arrows go in all directions

Pipeline-oriented
All arrows go left to right

A pipeline-oriented web backend
One directional flow,
even with branching

What are the benefits of
a pipeline-oriented approach?

Benefits
• Pipelines encourage composability
• Pipelines follow good design principles
• Pipelines are easier to maintain
• Pipelines make testing easier
• Pipelines fit well with modern architectures
– E.g. Onion/Clean/Hexagonal, etc

Benefit 1:
Pipelines encourage composability
Composable == “Like Lego”

Connect two pieces together and
get another "piece" that can still
be connected
You don't need to create a
special adapter to make
connections.

Component Component
Component
Pipelines composition
Pipelines encourage you to design so that:
• Any pieces can be connected
• You don’t need special adapters
If you do this, you get nice composable components

Bigger Component
Sub-
Component
Sub-
Component
Sub-
Component
Sub-components composed
into a new bigger unit

Benefit 2:
Pipelines follow
good design principles
Many design patterns work naturally
with a pipeline-oriented approach

Single responsibility principle
Component
Can't get more single responsibility than this!

Open/Closed principle
You could be able to add new functionality (“open for extension”)
without changing existing code (“closed for modification”)
New
Behavior
Component Component Component
Extension
Not changed

Strategy Pattern
Parameterized
Component
Component Component Component

Decorator Pattern
Decorator
Decorator
Original Component

Decorated Component
Decorator Pattern
Decorator
Decorator
"decorated" component works just like the original one

Benefit 3:
Pipelines are easier to maintain

This is easier to maintain
This is harder to maintain

Object-oriented: Real-world dependency graph
Multiple circular dependencies 
Arrows go in all directions

Pipeline-oriented: Real-world dependency graph
All arrows go left to right 
Same functionality
as the OO example

More benefits:
• Pipelines make testing easier
• Pipelines fit well with modern architectures

Pipeline-oriented Programming
in practice
Part 2

var count = 0;
foreach (var i in list) {
var j = i + 2;
if (j > 3)
{
count++;
}
}
return count;
Here is a
traditional
for- loop in C#

return list
.Select(x => x + 2)
.Where(x => x > 3)
.Count();
Here is the same code
using LINQ in C#

return list
.Select(x => x + 2)
.Where(x => x > 3)
.Count();
Benefit: Composability
The LINQ components have been designed
to fit together in many different ways
You could write your code this way too!

return list
.Select(x => x + 2)
.Where(x => x > 3)
.Count();
Benefit: Single responsibility
Each LINQ component does one thing only.
Easier to understand and test

return list
.Select(x => x + 2)
.Where(x => x > 3)
.Where(x => x < 10)
.Count();
Benefit: open for extension
It is easy to add new steps
in the pipeline without
touching anything else

list
|> List.map (fun x -> x + 2)
|> List.filter (fun x -> x > 3)
|> List.length
Here is the same code
using F#

Immutability and pipelines
If you have immutable data
you will probably need a pipeline

// Person is immutable
var p = new Person("Scott","s@example.com",21);
var p2 = p.WithName("Tom");
var p3 = p2.WithEmail("tom@example.com");
var p4 = p3.WithAge(42);
When each call returns
a new object, it gets
repetitive

// Person is immutable
var p = new Person("Scott","s@example.com",21);
p
.WithName("Tom")
.WithEmail("tom@example.com")
.WithAge(42); Pipelines make it
look nicer

Pipes vs. Extension methods
Pipes are more general

int Add1(int input) { return input + 1; }
int Square(int input) { return input * input; }
int Double(int input) { return input * 2; }

int Add1(int input) { return input + 1; }
int Square(int input) { return input * input; }
int Double(int input) { return input * 2; }
int NestedCalls()
{
return Double(Square(Add1(5)));
}
How can I make
this look nicer?
How about a
pipeline?

return 5
.Add1
.Square
.Double;
But this doesn't work,
because these are not
extension methods.

public static TOut Pipe<TIn, TOut>
(this TIn input, Func<TIn, TOut> fn)
{
return fn(input);
} Introducing "Pipe"

public static TOut Pipe<TIn, TOut>
(this TIn input, Func<TIn, TOut> fn)
{
return fn(input);
}

return 5
.Pipe(Add1)
.Pipe(Square)
.Pipe(Double);
And now we can create
a pipeline out of any*
existing functions

int Add(int i, int j) { return i + j; }
int Times(int i, int j) { return i * j; }
A two parameter
function

public static TOut Pipe<TIn, TParam, TOut>
(this TIn input, Func<TIn, TParam, TOut> fn, TParam p1)
{
return fn(input, p1);
}

int Add(int i, int j) { return i + j; }
int Times(int i, int j) { return i * j; }
public int PipelineWithParams()
{
return 5
.Pipe(Add, 1)
.Pipe(Times, 2);
}
We can now create a
pipeline out of any
existing functions
with parameters

{
return 5
.Pipe(Add, 1)
.Pipe(Times, 2);
}
Why bother?
Because now we get all the
benefits of a pipeline

{
return 5
.Pipe(Add, 1)
.Pipe(Times, 2)
.Pipe(Add, 42)
.Pipe(Square);
}
Why bother?
Because now we get all the
benefits of a pipeline,
such as adding things to
the pipeline easily
(diffs look nicer too!)

let add x y = x + y
let times x y = x * y
5
|> add 1
|> times 2
And here's what the same
code looks like in F#
F# uses pipelines
everywhere!

Pipes vs. Extension methods,
Part 2
Extension methods cannot be parameters

int StrategyPattern(... list, ... injectedFn) {
return list
.Select(x => x + 2)
.injectedFn
.Where(x => x > 3)
.Count();
} We cant use a function
parameter as an extension
method

My "strategy"

int StrategyPattern(... list, ... injectedFn) {
return list
.Select(x => x + 2)
.Pipe(injectedFn)
.Where(x => x > 3)
.Count();
} But we can use a function
parameter in a pipeline!


Three demonstrations
• Pipeline-oriented Roman Numerals
• Pipeline-oriented FizzBuzz
• Pipeline-oriented web API

To Roman Numerals
Task: convert an integer to roman numerals
• 5 => "V"
• 12 => "XII"
• 107 => "CVII"

To Roman Numerals
• Use the "tally" approach
– Start with N copies of "I"
– Replace five "I"s with a "V"
– Replace two "V"s with a "X"
– Replace five "X"s with a "L"
– Replace two "L"s with a "C"
– etc

To Roman Numerals
number
etc
Replicate "I"
Replace_IIIII_V
Replace_VV_X
Replace_XXXXX_L

string ToRomanNumerals(int n)
{
return new String('I', n)
.Replace("IIIII", "V")
.Replace("VV", "X")
.Replace("XXXXX", "L")
.Replace("LL", "C");
}
C# example

string ToRomanNumerals(int n)
{
return new String('I', n)
.Replace("IIIII", "V")
.Replace("VV", "X")
.Replace("XXXXX", "L")
.Replace("LL", "C");
}
C# example
.Replace("LL", "C")
.Replace("VIIII", "IX")
.Replace("IIII", "IV")
.Replace("LXXXX", "XC")
.Replace("XXXX", "XL");
}
Extend functionality
without touching
existing code

FizzBuzz definition
• Write a program that prints the numbers
from 1 to N
• But:
– For multiples of three print "Fizz" instead
– For multiples of five print "Buzz" instead
– For multiples of both three and five print
"FizzBuzz" instead.

for (var i = 1; i <= 30; i++)
{
if (i % 15 == 0)
Console.Write("FizzBuzz,");
else if (i % 3 == 0)
Console.Write("Fizz,");
else if (i % 5 == 0)
Console.Write("Buzz,");
else
Console.Write($"{i},");
}
C# example

Pipeline implementation
number Handle 15 case

Handle 3 case

Handle 3 case
Handle 5 case

Handle 3 case
Handle 5 case
number
Answer
Handle 15 case
Final step

number Handle case
Processed
(e.g. "Fizz", "Buzz")
Unprocessed
(e.g. 2, 7, 13)
record FizzBuzzData(string Output, int Number);

static FizzBuzzData Handle(
this FizzBuzzData data,
int divisor, // e.g. 3, 5, etc
string output) // e.g. "Fizz", "Buzz", etc
{
if (data.Output != "")
return data; // already processed
if (data.Number % divisor != 0)
return data; // not applicable
return new FizzBuzzData(output, data.Number);
}

static FizzBuzzData Handle(
this FizzBuzzData data,
int divisor, // e.g. 3, 5, etc
string output) // e.g. "Fizz", "Buzz", etc
{
if (data.Output != "")
return data; // already processed
if (data.Number % divisor != 0)
return data; // not applicable
return new FizzBuzzData(output, data.Number);
}
Extension method

static string FizzBuzzPipeline(int i)
{
return
new FizzBuzzData("", i)
.Handle(15, "FizzBuzz")
.Handle(3, "Fizz")
.Handle(5, "Buzz")
.FinalStep();
}
static void FizzBuzz()
{
var words = Enumerable.Range(1, 30)
.Select(FizzBuzzPipeline);
Console.WriteLine(string.Join(",", words));
}

Demo:
FizzBuzz in F#
Extensions, Decorations and
Parallelization

HttpContext Web Component
Not Handled
Handled

(async) HttpContext
null
HttpContext

GET
Define a component
matches verb
doesn't match verb
HttpContext

route "/hello"
Define a component
matches route
doesn't match route
HttpContext

setStatusCode 200
Define a component
Always succeeds
HttpContext

GET route "/hello"
>=>
A special pipe for web
components
setStatusCode 200
>=>

Are you wondering how do we compose these?

choose [
]
Picks first component
that succeeds
Define a new component

choose [
GET >=> route "/hello" >=> OK "Hello"
GET >=> route "/goodbye" >=> OK "Goodbye"
]
Pick first path
that succeeds

choose [
POST >=> route "/bad" >=> BAD_REQUEST
]
All the benefits of pipeline-oriented programming

choose [
POST
>=> route "/user"
>=> mustBeLoggedIn UNAUTHORIZED
>=> requiresRole "Admin"
// etc
]
All the benefits of pipeline-oriented programming

Demo:
A pipeline oriented web app
For more on the web framework I'm using,
search the internet for "F# Giraffe"

Testing and architecture
Part 4

Benefit 4:
Pipelines encourage
good architecture

The "onion" architecture
Core domain is pure, and all I/O is at the edges
Core
domain

Domain Logic
I/O I/O
The "onion" architecture

Core domain is pure, and all I/O is at the edges

In a well-designed pipeline,
all I/O is at the edges.
Easy to enforce this with a pipeline-oriented approach

Database Database
Load data
Process it
Save it

I/O in the middle of a pipeline
Keep them separate

Deterministic
Non-deterministic Non-deterministic

Why bother?
• Reusable components
• Understandable – data flows in one direction
• Extendable – add new parts without touching old
code
• Testable – parts can be tested in isolation
• A different way of thinking –
it's good for your brain to learn new things!

Databases
Object-oriented
programmng
Pipeline-oriented
programming
Domain-driven
design
Does pipeline-oriented programming
work for every situation? No!
But it should be part of your toolkit

Pipeline Oriented Programming
– Slides and video will be posted at
• fsharpforfunandprofit.com/pipeline
Related talks
– "The Power of Composition"
• fsharpforfunandprofit.com/composition
– “Railway Oriented Programming"
• fsharpforfunandprofit.com/rop
Thanks!
Twitter: @ScottWlaschin

Pipeline oriented programming

Recommended

More Related Content

What's hot (20)

Similar to Pipeline oriented programming (20)

More from Scott Wlaschin (14)

Recently uploaded (20)

Pipeline oriented programming