C# functional language extensions - a base class library for functional programming
WARNING: THIS IS AN ALPHA RELEASE AND SHOULD BE CONSUMED WITH CARE! NOT FOR PRODUCTION.
For those that don't know yet (and there's no reason to think you should, because I haven't announced it yet) -- the Pipes Effect system now has the ability to lift any monad into its stack (previously it only allowed Aff to be lifted). It is now a general monad transformer like ReaderT, OptionT, EitherT, etc.
As, with all monad-transfomers, when you 'run' the transformer, it generates the lifted monad. You can think of this being like a mini-compiler that takes the monad stack and compiles down to the inner-most monad, which can then just be run as normal.
The problem for Pipes is that there's usually lots of recursion, repetition (using repeat, retry), or iteration (using yieldAll, etc.). This is problematic when you don't know anything about the inner monad. The transformer can't run the inner monad, because it only has access to the Monad interface (Bind) and the inherited interfaces of Applicative and Functor (Apply, Action, Map, and Pure). So, doing iteration requires recursion, and recursion blows the stack in C#.
Previously Pipes were able to directly
RuntheAffbecause the Pipe system knew it was working only withAff. This allowed it to flatten the recursion.
Anyway, now Pipes has internal support for any Foldable. That means yieldAll(...) can take a sequence from any foldable (Range, EnumerableM, HashMap, HashSet, Lst, Map, Seq, Either, Option, Validation, Identity, ... and any you write) and yield the values within the structure through the pipe. Functions like repeat(ma) - which continually repeat an operation until it fails - have also been implemented internally as something that iterates over an infinite foldable.
This functionality has been enabled by adding a new method to the Applicative trait: Actions. You might know the existing Action(K<M, A> ma, K<M, B> mb) method that runs the first applicative (ma), ignores its result, and then runs the second applicative mb, returning its result.
Actions instead takes an IEnumerable<K<M, A>>:
K<F, A> Actions<A>(IEnumerable<K<F, A>> fas)
It runs each applicative action and ignores its result, returning the result of the last item. That means a sequence of Proxy values (Proxy is the monad-transformer for pipes) can be mapped - the map will just run (using RunEffect) the Proxy - producing a sequence of whatever the lifted inner-monad is for the Proxy. This lazy sequence of monads can then be invoked by calling Actions on it, which will lazily walk the sequence, evaluating the inner-monad one-by-one.
There is a default implementation, but it has the same lack of knowledge that Pipes had, so it should be overridden for computation based applicatives (that usually need invoking with without an argument). Here's the override for Eff<RT, A>:
static K<Eff<RT>, A> Applicative<Eff<RT>>.Actions<A>(IEnumerable<K<Eff<RT>, A>> fas) =>
from s in getState<A>()
from r in Eff<RT, A>.Lift(
rt =>
{
Fin<A> rs = Errors.SequenceEmpty;
foreach (var kfa in fas)
{
var fa = kfa.As();
rs = fa.Run(rt, s.Resources, s.EnvIO);
if (rs.IsFail) return rs;
}
return rs;
})
select r;
You can see how:
And so, if you want to use your own monads with Pipes then you should implement Actions.
There's still more to do with Pipes, but all of the examples in EffectsExamples now work, which is a good sign!
WARNING: THIS IS AN ALPHA RELEASE AND SHOULD BE CONSUMED WITH CARE! NOT FOR PRODUCTION.
WARNING: THIS IS AN ALPHA RELEASE AND SHOULD BE CONSUMED WITH CARE! NOT FOR PRODUCTION.
Free monad doesn't need Alternative trait: removedAppend operator renamed to Combine. 'Combine' works semantically for more of the monoidal associative operations than Append (which really only makes sense with collections).SemigroupK and MonoidK -- these are like the Semigroup and Monoid traits except they work on K<M, A> instead of A. These are almost identical to SemiAlternative and Alternative execept they don't require the underlying value to an an Applicative. The idea here is that SemigroupK and MonoidK would be used on types like collections that 'sum' when the Combine operator is applied, whereas SemiAlternative and Alternative provide an alternative value when the Combine operator is applied (coalescing).repeat variants, retry variants, and timeout for the IO monadIO.yieldFor(TimeSpan). This is like Task.Delay but for the IO monad. The war against async means that this does the thread-yielding internally, no need to call await. I figured yieldFor is more meaningful than Delay, it indicates that the thread is yielding, not simply blocking.ContT<R, M, A> -- just the raw type for now.HeadOrNone, HeadOrInvalid, HeadOrLeft, LastOrNone, etc. have been removed.Head and Last are now Option returning. This is a breaking change. Can be mitigated by either matching, casting, or invocation of .ValueUnsafe() extension.Range type -- previously there were several types (IntegerRange, CharRange, etc.) -- now there's just one: Range<A>. It leverages the new traits built into .NET (IComparisonOperators, IAddtionOperators, etc.)Foldable.Sum, Foldable.Max, etc.: (IComparisonOperators, IAddtionOperators, IAdditiveIdentity etc.) -- these are Microsoft's ugly named versions of monoids etc.I'm rapidly coming to the conclusion that extension-methods are a terrible idea. Especially in a library like language-ext where I am trying to present a consistent set of interfaces to types that share common traits. It's just impossible to enforce consistency and relies on the human eye -- and that errs regularly!
The latest move toward using traits is really starting to help reduce the extension methods, or at least mean the extension methods are hanging off traits rather than individual instance-types.
One change that I have made recently is to change Foldable to require implementation of FoldWhile and FoldWhileBack instead of Fold and FoldBack. This means that so many more default behaviours can hang off of Foldable -- and most of them are optimal. For example, Exists -- which can stop processing as soon as its predicate returns true -- couldn't early-out before.
And so, the foldable trait is now growing to have a ton of functionality. Also nested foldables!
However, quite a lot of those methods, like Sum, Count, etc. also exist on IEnumerable. And so, for a type like Seq which derives from both IEnumerable and K<Seq, A>, there will be extension method resolution issues.
So, the choice is to provide extension methods for IEnumerable (an ill defined type) or for Foldable - a super featureful type with the opportunity for implementers to provide bespoke optimised overrides.
Really, the choice should be easy: extensions for Foldable are just better than extensions for IEnumerable. So, I have done that. The downside is that this will be another breaking change (because the IEnumerable extensions have been removed). The fix is to convert from IEnumerable<A> to EnumerableM<A> using .AsEnumerableM(). EnumerableM<A> supports Foldable (and other traits).
So, I've been working to remove as many non-trait extension methods as I can -- and I will continue to do so leading up to the beta. This will bring consistency to the code-base, reduce the amount of code, and provide ample opportunities for bespoke optimisations. Just be aware that this is another fix-up job.
WARNING: THIS IS AN ALPHA RELEASE AND SHOULD BE CONSUMED WITH CARE! NOT FOR PRODUCTION.
This release should only be consumed by those who are interested in the new features coming in the monster v5 release.
Just to give you an idea of the scale of this change:
It is a monster and should be treated with caution...
If you add it to a production project, you should only do so to see (potentially) how many breaking changes there are. I would not advise migrating a production code-base until I get close to the final release.
I am also not going to go into huge detail about the changes here, I will simply list them as headings. I will do a full set of release notes for the beta release. You can however follow the series of articles I am writing to help you all prep for v5 -- it goes (and will go) into much more detail about the features.
K<F, A> - higher-kinds enabling interfaceFunctor.map, Alternative.or, StateM.get, ...).Map, .Or, Bind, etc.),BindT, MapT, etc. ), now fully generic.Option, Either<L>, etc.)Functor<F>
Applicative<F>
Monad<M>
Foldable<F>
Traversable<T>
Alternative<F>
SemiAlternative<F>
Has<M, TRAIT>
Reads<M, OUTER_STATE, INNER_STATE>
Mutates<M, OUTER_STATE, INNER_STATE>
ReaderM<M, Env>
StateM<M, S>
WriterM<M, OUT>
MonadT<M, N> - Monad transformers
ReaderT<Env, M, A>
WriterT<Out, M, A>
StateT<S, M, A>
IdentityT<M, A>
EitherT<L, M, R>
ValidationT<F, M, S>
OptionT<M, A>
TryT<M, A>
IdentityT<M, A>
ResourceT<M, A>
Free<F, A> - Free monadsIO<A> - new IO monad that is the base for all IOEff<RT, A> monad rewritten to use monad-transformers (StateT<RT, ResourceT<IO>, A>)Eff<RT, A> doesn't need HasCancel trait (or any trait)Pure / Fail monadsSeq1 made [Obsolete]
Semigroup<A> and Monoid<A> types have been refactoredTypeClass class has been renamed Trait
Apply extensions that use raw Func removedSequence extension methods have been removedTraverse extension methods have been removedToComparer doesn't exist on the Ord<A> trait any moreLanguageExt.ClassInstances.Sum
Guard<E> has become Guard<E, A>
UnitsOfMeasaure namespace converted to a static classEither doesn't support IEnumerable<EitherData> any moreEither 'bi' functions have their arguments flippedTuple and KeyValuePair removedSome<A>
OptionNone
EitherUnsafe<L, R>
EitherLeft<L>
EitherRight<L>
Validation<MFail, Fail, A>
Try<A>
TryOption<A>
TryAsync<A>
TryOptionAsync<A>
Result<A>
OptionalResult<A>
Option<A>
ExceptionMatch, ExceptionMatchAsync, ExceptionMatchOptionalAsync
LanguageExt.SysX
LanguageExt.CodeGen
LanguageExt.Transformers
New:
IfFail in Try, TryOption, TryAsync, and TryOptionAsync - Thanks @mark-pro 👍Effects samplesBug fixes:
This is a fixes release.
OptionAsyncI have brought forward a change to OptionAsync that I was saving for v5: the removal of the async-awaiter. You can't now await an OptionAsync. The resulting value wasn't clear, and honestly the async/await machinery is really quite shonky outside of using it for Tasks.
I have made the OptionAsync implementation aware of nullable references, and so you can now await the Value property instead:
public Task<A?> Value
That will reproduce the same behaviour as before. You can still await the ToOption() method, which returns a Task<Option<A>>, if you want to do matching on the underlying option. Or call the various Match* methods.
This release fixes the following issues:
Producer.merge error handlingProducer merging was silently ignoring errors. They now exit and return the first error and shutdown other producers they were merged with. Merged producers also listen for cancellation correctly now.
Finally, you can only merge produces with a bound value of Unit. This is to stop the silent dropping of their return value as well as the need to provide a final (bound) value for merged producers, which doesn't really make sense. That also means the + operator can't work any more because it can't be defined for the Producer<..., A> type. So you must use Producer.merge.
This fixes an issue mentioned in: https://github.com/louthy/language-ext/issues/1177
repeatM doesn't cause a stack-overflowCertain elements of the Pipes capability of language-ext are direct ports from the Haskell Pipes library, which uses recursion everywhere. repeatM was causing a stack-overflow on usage, this is now fixed.
Example usage:
public static Effect<Runtime, Unit> effect =>
Producer.repeatM(Time<Runtime>.nowUTC) | writeLine<DateTime>();
static Consumer<Runtime, X, Unit> writeLine<X>() =>
from x in awaiting<X>()
from _ in Console<Runtime>.writeLine($"{x}")
from r in writeLine<X>()
select r;
repeat improvementsRemoved the Repeat case from the Pipes DSL which simplifies it and brings it closer to the Haskell version. Updated the repeat combinator function to use the same Enumerate case that yieldAll uses. This has benefits that it doesn't spread out when composed with other Proxy types. This is should mean it's easier to pick bits of the expression to repeat, rather than the whole effect being repeated due to the spread.
TrampolineAdded trampolining functionality. It's relatively light at the moment, I am considering approaches to enable genuine recursion in the effects system. Don't rely on this, it may be removed if it doesn't prove useful and almost certainly will have API changes if it stays.
There Pipes functions: enumerate, enumerate2, observe, observe2 have been deleted and replaced with yieldAll (that accepts IEnumerable, IAsyncEnumerable, or IObservable).
The previous implementation had mixed behaviours, some that always yielded the values, some that turned the remainder of the pipes expression into a enumeration. This wasn't entirely clear from the name and so now there is a single set of yieldAll functions that always yield all the values in the collection downstream.
The behaviour of the always yield enumerate functions was also buggy, and didn't result in the remainder of a Producer or Pipe being invoked after the yield. :
public static Effect<Runtime, Unit> effect =>
repeat(producer) | consumer;
static Producer<Runtime, int, Unit> producer =>
from _1 in Console<Runtime>.writeLine("before")
from _2 in yieldAll(Range(1, 5))
from _3 in Console<Runtime>.writeLine("after")
select unit;
static Consumer<Runtime, int, Unit> consumer =>
from i in awaiting<int>()
from _ in Console<Runtime>.writeLine(i.ToString())
select unit;
In the example above, "after" would never be called, this is now fixed.
There is also a new & operator overload for Pipes which performs the operations in series. This has the effect of concatenating Producers (for example), but will work for Pipe, Consumer, Client, and Server.
// yields [1..10]
static Producer<Runtime, int, Unit> producer =>
yieldAll(Range(1, 5)) & yieldAll(Range(6, 5));
There's still work to do on repeat, but this was quite a difficult change, so I'll leave that for now.
This release puts out the 4.3.* beta changes:
And contains a number of contributed improvements:
And a number of contributed bug fixes:
Task.Cast and ValueTask.Cast
Thanks to all those who contributed. I am still super busy with other projects right now, and I don't always get to PRs as quickly as I would like, but It's always appreciated.
Any problems, please report in the Issues.
There have been a number of calls on the Issues page for a ValidationAsync monad, which although it's a reasonable request (and I'll get to it at some point I'm sure), when I look at the example requests, it seems mostly the requestors want a smarter error handling story in general (especially for the collection of multiple errors).
The error-type that I'm building most of the modern functionality around (in Fin, Aff, and Eff for example) is the struct type: Error. It has been designed to handle both exceptional and expected errors. But the story around multiple errors was poor. Also, it wasn't possible to carry additional information with the Error, it was a closed-type other than ability to wrap up an Exception - so any additional data payloads was cumbersome and ugly.
Extending the
structtype to be more featureful was asking for trouble, as it was already getting pretty messy.
Error refactorSo, I've bitten the bullet and refactored Error into an abstract record type.
Error sub-typesThere are a few built-in sub-types:
Exceptional - An unexpected errorExpected - An expected errorManyErrors - Many errors (possibly zero)These are the key base-types that indicate the 'flavour' of the error. For example, a 'user not found' error isn't
something exceptional, it's something we expect to happen. An OutOfMemoryException however, is
exceptional - it should never happen, and we should treat it as such.
Most of the time we want sensible handling of expected errors, and bail out completely for something exceptional. We also want to protect ourselves from information leakage. Leaking exceptional errors via public APIs is a sure-fire way to open up more information to hackers than you would like. The Error derived types all try to protect against this kind of leakage without losing the context of the type of error thrown.
When Exceptional is serialised, only the Message and Code component is serialised. There's no serialisation of the inner Exception or its stack-trace. It is also possible to construct an Exceptional message with an alternative message:
Error.New("There was a problem", exception);
That means if the Error gets serialised, we only get a "There was a problem" and an error-code.
Deserialisation obviously means we can't recover the
Exception, but the state of theErrorwill still beExceptional- so it's possible to carry the severity of the error across domain boundaries without leaking too much information.
Error methods and propertiesEssentially an error is either created from an Exception or it isn't. This allows for expected errors to be represented without throwing exceptions, but also it allows for more principled error handling. We can pattern-match on the
type, or use some of the built-in properties and methods to inspect the Error:
IsExceptional - true for exceptional errors. For ManyErrors this is true if any of the errors are exceptional.IsExpected - true for non-exceptional/expected errors. For ManyErrors this is true if all of the errors are expected.Is<E>(E exception) - true if the Error is exceptional and any of the the internal Exception values are of type E.Is(Error error) - true if the Error matches the one provided. i.e. error.Is(Errors.TimedOut).IsEmpty - true if there are no errors in a ManyErrors
Count - 1 for most errors, or n for the number of errors in a ManyErrors
Head() - To get the first errorTail() - To get the tail of multiple errorsYou may wonder why
ManyErrorscould be empty. That allows forErrors.None- which works a little likeOption.None. We're saying: "The operation failed, but we have no information on why; it just did".
Error constructionThe Error type can be constructed as before, with the various overloaded Error.New(...) calls.
For example, this is an expected error:
Error.New("This error was expected")
When expected errors are used with codes then equality and matching is done via the code only:
Error.New(404, "Page not found");
And this is an exceptional error:
try
{
}
catch(Exception e)
{
// This wraps up the exceptional error
return Error.New(e);
}
Finally, you can collect many errors:
Error.Many(Error.New("error one"), Error.New("error two"));
Or more simply:
Error.New("error one") + Error.New("error two")
Error types with additional dataYou can extend the set of error types (perhaps for passing through extra data) by creating a new record that inherits Exceptional or Expected:
public record BespokeError(bool MyData) : Expected("Something bespoke", 100, None);
By default the properties of the new error-type won't be serialised. So, if you want to pass a payload over the wire, add the [property: DataMember] attribute to each member:
public record BespokeError([property: DataMember] bool MyData) : Expected("Something bespoke", 100, None);
Using this technique it's trivial to create new error-types when additional data needs to be moved around, but also there's a ton of built-in functionality for the most common use-cases.
Error breaking changesError isn't a struct any more, default(Error) will now result in null. In practice this shouldn't affect anyone.BottomException is now in LanguageExt.Common
Error documentationThere's also a big improvement on the API documentation for the Error types
Aff and Eff applicative functorsNow that Error can handle multiple errors, we can implement applicative behaviours for Aff and Eff. If you think of monads enforcing sequential operations (and therefore can only continue if each operation succeeds - leading to only one error report if it fails), then applicative-functors are the opposite in that they can run independently.
This is what's used for the
Validationmonads, to allow multiple operations to be evaluated, and then all of the errors collected.
By adding Apply to Aff and Eff, we can now do the same kind of validation-logic both synchronously and asynchronously.
First let's create a simple asynchronous effect that delays for a period of time:
static Aff<Unit> delay(int milliseconds) =>
Aff(async () =>
{
await Task.Delay(milliseconds);
return unit;
});
Now we'll combine that so we get an effect that parses a string into an int, and adds a delay of 1000 milliseconds (the delay is to simulate calling some external IO).
:
static Aff<int> parse(string str) =>
from x in parseInt(str).ToAff(Error.New("parse error: expected int"))
from _ in delay(1000)
select x;
Notice how we're converting the
Option<int>to anAff, and providing an error value to use if theOptionisNone
Next we'll use the applicative behaviour of the Aff to run two operations in parallel. When they complete the values will be applied to the function that has been lifted by SuccessAff.
static Aff<int> add(string sx, string sy) =>
SuccessAff((int x, int y) => x + y)
.Apply(parse(sx), parse(sy));
To measure what we're doing, let's add a simple function called report. All it does is run an Aff, measures how long it takes, and prints the results to the screen:
static async Task report<A>(Aff<A> ma)
{
var sw = Stopwatch.StartNew();
var r = await ma.Run();
sw.Stop();
Console.WriteLine($"Result: {r} in {sw.ElapsedMilliseconds}ms");
}
Finally, we can run it:
await report(add("100", "200"));
await report(add("zzz", "yyy"));
The output for the two operations is this:
Result: Succ(300) in 1032ms
Result: Fail([parse error: expected int, parse error: expected int]) in 13ms
Notice how the first one (which succeeds) takes 1032ms - i.e. the two parse operations ran in parallel. And on the second one, we get both of the errors returned. The reason that one finished so quickly is because the delay was after the parseInt call, so we exited immediately.
Of course, it would be possible to do this:
from x in parse(sx)
from y in parse(sy)
select x + y;
Which is more elegant. But the success path would take 2000ms, and the failure path would only report the first error.
Hopefully that gives some insight into the power of applicatives (even if they're a bit ugly in C#!)
This will be in beta for a little while, as the changes to the Error type are not trivial.
The existing Schedule type has been massively upgraded to support even more complex scheduling for repeating, retrying, and folding of Aff and Eff types.
A huge thanks to @bmazzarol who did all of the heavy lifting to make this feature a reality!
It has been refactored from the ground up, a Schedule now is a (possibly infinite) stream of durations. Each duration indicates to the retry, repeat, and fold behaviours how long to wait between each action. The generation of those streams comes from:
Schedule.Forever - infinite stream of zero length durationsSchedule.Once - one item stream of zero length durationSchedule.Never - no durations (a schedule that never runs)Schedule.TimeSeries(1, 2, 3 ...) - pass in your own durations to build a bespoke scheduleSchedule.spaced(space) - infinite stream of space length durationsSchedule.linear(seed, factor) - schedule that recurs continuously using a linear back-offSchedule.exponential(seed, factor) - schedule that recurs continuously using a exponential back-offSchedule.fibonacci(seed, factor) - schedule that recurs continuously using a fibonacci based back-offSchedule.upto(max) - schedule that runs for a given durationSchedule.fixedInterval(interval) - if that action run between updates takes longer than the interval, then the action will run immediatelySchedule.windowed(interval) - a schedule that divides the timeline into interval-long windows, and sleeps until the nearest window boundary every time it recurs.Schedule.secondOfMinute(second) - a schedule that recurs every specified second of each minuteSchedule.minuteOfHour(minute) - a schedule that recurs every specified minute of each hourSchedule.hourOfDay(hour) - a schedule that recurs every specified hour of each daySchedule.dayOfWeek(day) - a schedule that recurs every specified day of each weekThese schedules are mostly infinite series, and so to control their 'length' we compose with ScheduleTransformer values to create smaller series, or to manipulate the series in some way (jitter for example). The following functions generate ScheduleTransformer values.
Schedule.recurs(n) - Clamps the schedule durations to only recur n times.Schedule.NoDelayOnFirst - Regardless of any other settings, it makes the first duration zeroSchedule.RepeatForever - Repeats any composed schedules foreverSchedule.maxDelay(max) - limits the returned delays to max delay (upper clamping of durations).Schedule.maxCumulativeDelay(Duration max) - keeps a tally of all the delays so-far, and ends the generation of the series once max delay has passedSchedule.jitter(minRandom, maxRandom, seed) - adds random jitter to the durationsSchedule.jitter(factor, seed) - adds random jitter to the durationsSchedule.decorrelate(factor, seed) - transforms the schedule by de-correlating each of the durations both up and down in a jittered way.Schedule.resetAfter(max) - resets the schedule after a provided cumulative max durationSchedule.repeats(n) - not to be confused with recurs, this repeats the schedule n times.Schedule.intersperse(schedule) - intersperse the provided schedule between each duration in the schedule.Schedule and ScheduleTransformer can be composed using | (union) or & (intersection):
var schedule = Schedule.linear(1 * sec) | Schedule.recurs(3) | Schedule.repeat(3);
// [1s, 2s, 3s, 1s, 2s, 3s, 1s, 2s, 3s]
Union | will take the minimum of the two schedules to the length of the longest, intersect & will take the maximum of the two schedules to the length of the shortest.
One thing remaining to-do is to bring
HasTime<RT>back into theCoreand allow these schedules to use injectable time. Some of the functions already take aFunc<DateTime>to access 'now', this will be expanded so time can be sped up or slowed down, with the schedules 'just working'. That'll be in the next few weeks I'm sure, and is related to this issue.
Check out the API documentation to see what's what. And again, thanks to @bmazzarol for the hard work 👍
The transformers extensions, which are a big set of T4 templates for generating extension methods for nested monadic types have now been broken out into their own package: LanguageExt.Transformers
If you use the following functions: BindT, MapT, FoldT, FoldBackT, ExistsT, ForAllT, IterT, FilterT, PlusT, SubtractT, ProductT, DivideT, SumT, CountT, AppendT, CompareT, EqualsT, or ApplyT - then you will get compile errors, and will need to add a reference to the LanguageExt.Transformers package.
I've done this for a couple of reasons:
LanguageExt.Core library. This change takes the Core package from 3,276 kb to 2,051 kb.
Core library will always be quite chunky because of the sheer amount of features, but the transformer extension methods definitely aren't always needed, so breaking them out made senseThe main transformer extensions that remain in the Core library are:
Traverse
Sequence
These are so heavily used that I believe moving them out into the Transformers library would mean everyone would be obliged to use it, and therefore it wouldn't achieve anything. There may be an argument for bringing BindT and MapT back into the core at some point. I will see how this plays out (it wouldn't be a future breaking change if that were the case).
Any problems, please report via the Issues in the usual way.