a new concurrent ml

29 June 2017 2:37 PM (guile | scheme | gnu | igalia | compilers | delimited continuations | prompts | fibers | concurrency | concurrent ml | curry on)

Good morning all!

In my last article I talked about how we composed a lightweight "fibers" facility in Guile out of lower-level primitives. What we implemented there is enough to be useful, but it is missing an important aspect of concurrency: communication. Sure, being able to spawn off fibers is nice, but you have to be able to actually talk to them.

Fibers had just gotten to the state described above about a year ago as I caught a train from Geneva to Rome for Curry On 2016. Train rides are magnificent for organizing thoughts, and I was in dire need of clarity. I had tentatively settled on Go-style channels by the time I got there, but when I saw that Matthias Felleisen and Matthew Flatt there, I had to take advantage of the opportunity to ask them what they thought. Would they recommend Racket-like threads and channels? Had that been a good experience?

The answer that I got in return was a "yes, that's what you should do", but also a "you should look at Concurrent ML". Concurrent ML? What's that? I looked and was a bit skeptical. It seemed old and hoary and maybe channels were just as expressive. I looked more deeply into this issue and it seemed CML is a bit more expressive than just channels but damn, it looked complicated to implement.

I was wrong. This article shows that what you need to do to implement multi-core CML is actually the same as what you need to do to implement channels in a multi-core environment. By building CML first and channels and whatever later, you get more power for the same amount of work.

Note that this article has an associated talk! If video is your thing, see my Curry On 2017 talk here:

Or, watch on the youtube if the inline video above doesn't work; slides here as well.

on channels

Let's first have a crack at implementing channels. Before we begin, we should be a bit more explicit about what a channel is. My first hack in this area did the wrong thing: I was used to asynchronous queues, and I thought that's what a channel was. Besides ignorance, apparently that's what Erlang does; a process's inbox is an unbounded queue of messages with only very slight back-pressure.

But an asynchronous queue is not a channel, at least in its classic sense. As they were originally formulated in "Communicating Sequential Processes" by Tony Hoare, adopted into David May's occam, and from there into many other languages, channels are meeting-places. Processes meet at a channel to exchange values; whichever party arrives first has to wait for the other party to show up. The message that is handed off in a channel send/receive operation is never "owned" by the channel; it is either owned by a sender who is waiting at the meeting point for a receiver, or it's accepted by a receiver. After the transaction is complete, both parties continue on.

You'd think this is a fine detail, but meeting-place channels are strictly more expressive than buffered channels. I was actually called out for this because my first implementation of channels for Fibers had effectively a minimum buffer size of 1. In Go, whose channels are unbuffered by default, you can use a channel for RPC:

package main

func double(ch chan int) {
  for { ch <- (<-ch * 2) }

func main() {
  ch := make(chan int)
  go double(ch)
  ch <- 2
  x := <-ch

Here you see that the main function sent a value on ch, then immediately read a response from the same channel. If the channel were buffered, then we'd probably read the value we sent instead of the doubled value supplied by the double goroutine. I say "probably" because it's not deterministic! Likewise the double routine could read its responses as its inputs.

Anyway, the channels we are looking to build are meeting-place channels. If you are interested in the broader design questions, you might enjoy the incomplete history of language facilities for concurrency article I wrote late last year.

With that prelude out of the way, here's a first draft at the implementation of the "receive" operation on a channel.

(define (recv ch)
  (match ch
    (($ $channel recvq sendq)
     (match (try-dequeue! sendq)
       (#(value resume-sender)
         (lambda (k)
           (define (resume val)
             (schedule (lambda () (k val)))
           (enqueue! recvq resume))))))))

;; Note: this code has a race!  Fixed later.

A channel is a record with two fields, its recvq and sendq. The receive queue (recvq) holds a FIFO queue of continuations that are waiting to receive values, and the send queue holds continuations that are waiting to send values, along with the value that they are sending. Both the recvq and the sendq are lockless queues.

To receive a value from a meeting-place channel, there are two possibilities: either there's a sender already there and waiting, or we have to wait for a sender. Those two cases are handled above, in that order. We use the suspend primitive from the last article to arrange for the fiber to wait; presumably the sender will resume us when they arrive later at the meeting-point.

an aside on lockless data structures

We'll go more deeply into the channel receive mechanics later, but first, a general question: what's the right way to implement a data structure that can be accessed and modified concurrently without locks? Though I am full of hubris, I don't have enough to answer this question definitively. I know many ways, but none that's optimal in all ways.

For what I needed in Fibers, I chose to err on the side of simplicity.

Some data in Fibers is never modified; this immutable data is safe to access concurrently from any code. This is the best, obviously :)

Some mutable data is only ever mutated from an "owner" core; it's safe to read without a lock from that owner core, and in Fibers we do not access this data from other cores. An example of this kind of data structure is the i/o map from file descriptors to continuations; it's core-local. I say "core-local" because in fibers we typically run one scheduler per core, with each core having a pinned POSIX thread; it's really thread-local but I don't want to use the word "thread" too much here as it's confusing.

Some mutable data needs to be read and written from many cores. An example of this is the recvq of a channel; many receivers and senders can be wanting to read and write there at once. The approach we take in Fibers is just to use immutable data stored inside an "atomic box". An atomic box holds a single value, and exposes operations to read, write, swap, and compare-and-swap (CAS) the value. To read a value, just fetch it from the box; you then have immutable data that you can analyze without locks. Having read a value, you can to compute a new state and use CAS on the atomic box to publish that change. If the CAS succeeds, then great; otherwise the state changed in the meantime, so you typically want to loop and try again.

Single-word CAS suffices for Guile when every value stored into an atomic box will be unique, a property that freshly-allocated objects have and of which GC ensures us an endless supply. Note that for this to work, the values can share structure internally but the outer reference has to be freshly allocated.

The combination of freshly-allocated data structures and atomic variables is a joy to use: no hassles about multi-word compare-and-swap or the ABA problem. Assuming your GC can keep up (Guile's does about 700 MB/s), it can be an effective strategy, and is certainly less error-prone than others.

back at the channel recv ranch

Now, the theme here is "growing a language": taking primitives and using them to compose more expressive abstractions. In that regard, sure, channel send and receive are nice, but what about select, which allows us to wait on any channel in a set of channels? How do we take what we have and built non-determinism on top?

I think we should begin by noting that select in Go for example isn't just about receiving messages. You can select on the first channel that can send, or between send and receive operations.

select {
case c <- x:
  x, y = y, x+y
case <-quit:

As you can see, Go provides special syntax for select. Although in Guile we can of course provide macros, usually those macros expand out to a procedure call; the macro is sugar for a function. So we want select as a function. But because we need to be able to select over receiving and sending at the same time, the function needs to take some kind of annotation on what we are going to do with the channels:

(select (recv A) (send B v))

So what we do is to introduce the concept of an operation, which is simply data describing some event which may occur in the future. The arguments to select are now operations.

(select (recv-op A) (send-op B v))

Here recv-op is obviously a constructor for the channel-receive operation, and likewise for send-op. And actually, given that we've just made an abstraction over sending or receiving on a channel, we might as well make an abstraction over choosing the first available op among a set of operations. The implementation of select now creates such a choice-op, then performs it.

(define (select . ops)
  (perform (apply choice-op ops)))

But what we're missing here is the ability to know which operation actually happened. In Go, select's special syntax associates a clause of code with each sub-operation. In Scheme a clause of code is just a function, and so what we want to do is to be able to annotate an operation with a function that will get run if the operation succeeds.

So we define a (wrap-op op k), which makes an operation that itself annotates op, associating it with k. If op occurs, its result values will be passed to k. For example, if we make a fiber that tries to perform this operation:

  (recv-op A)
  (lambda (v)
    (string-append "hello, " v))))

If we send the string "world" on the channel A, then the result of this perform invocation will be "hello, world". Providing "wrapped" operations to select allows us to handle the various cases in separate, appropriate ways.

we just made concurrent ml

Hey, we just reinvented Concurrent ML! In his PLDI 1988 paper "Synchronous operations as first-class values", John Reppy proposes just this abstraction. I like to compare it to the relationship between an expression (exp) and wrapping that expression in a lambda ((lambda () exp)); evaluating an expression gives its value, and the expression just goes away, whereas evaluating a lambda gives a procedure that you can call in the future to evaluate the expression. You can call the lambda many times, or no times. In the same way, a channel-receive operation is an abstraction over receiving a value from a channel. You can perform that operation many times, once, or not at all.

Reppy consolidated this work in his PLDI 1991 paper, "CML: A higher-order concurrent language". Note that he uses the term "event" instead of "operation". To me the name "event" to apply to this abstraction never felt quite right; I guess I wrote too much code in the past against event loops. I see "events" as single instances in time and not an abstraction over the possibility of a, well, of an event. Indeed I wish I could refer to an instantiation of an operation as an event, but better not to muddy the waters. Likewise Reppy uses "synchronize" where I use "perform". As you like, really, it's still Concurrent ML; I just prefer to explain to my users using terms that make sense to me.

what's an op?

Let's return to that channel recv implementation. It had basically two parts: an optimistic part, where the operation could complete immediately, and a pessimistic part, where we had to wait for the other party to arrive. However, there was a race condition, as I noted in the comment. If a sender and a receiver concurrently arrive at a channel, it could be that they concurrently do the optimistic check, don't notice that the other is there, then they both suspend, waiting for each other to arrive: deadlock. To fix this for recv, we have to recheck the sendq after publishing our presence to the recvq.

I'll get to the details in a bit for channels, but it turns out that this is a general pattern. All kinds of ops have optimistic and pessimistic behavior.

(define (perform op)
  (match op
    (($ $op try block wrap)
     (define (do-op)
       ;; Return a thunk that has result values.
       (or optimistic
     ;; Return values, passed through wrap function.
     ((compose wrap do-op)))))

In the optimistic phase, the calling fiber will try to commit the operation directly. If that succeeds, then the calling fiber resumes any other fibers that are part of the transaction, and the calling fiber continues. In the pessimistic phase, we park the calling fiber, publish the fact that we're ready and waiting for the operation, then to resolve the race condition we have to try again to complete the operation. In either case we pass the result(s) through the wrap function.

Given that the pessimistic phase has to include a re-check for operation completability, the optimistic phase is purely an optimization. It's a good optimization that everyone will want to implement, but it's not strictly necessary. It's OK for a try function to always return #f.

As shown in the above function, an operation is a plain old data structure with three fields: a try, a block, and a wrap function. The optimistic behavior is implemented by the try function; the pessimistic side is partly implemented by perform, which handles the fiber suspension part, and by the operation's block function. The wrap function implements the wrap-op behavior described above, and is applied to the result(s) of a successful operation.

Now, it was about at this point that I was thinking "jeebs, this CML thing is complicated". I was both wrong and right -- there's some complication inherent in multicore lockless communication, yes, but I believe CML captures something close to the minimum, and certainly it's just as much work as with a direct implementation of channels. In that spirit, I continue on with the implementation of channel operations in Fibers.

channel receive operation

Here's an implementation of a try function for a channel.

(define (try-recv ch)
  (match ch
    (($ $channel recvq sendq)
     (let ((q (atomic-ref sendq)))
       (match q
         (() #f)
         ((head . tail)
          (match head
            (#(val resume-sender state)
             (match (CAS! state 'W 'S)
                (resume-sender (lambda () (values)))
                (CAS! sendq q tail) ; *
                (lambda () val))
               (_ #f))))))))))

In Fibers, a try function either succeeds and returns a thunk, or fails and returns #f. For channel receive, we only succeed if there is a sender already in the queue: the sender has arrived, suspended itself, and published its availability. The state variable is an atomic box that holds the operation state, which initially starts as W and when complete is S. More on that in a minute. If the CAS! compare-and-swap operation managed to change the state from W to S, then the optimistic phase suceeded -- yay! We resume the sender with no values, take the value that the sender gave us, and keep on trucking, returning that value wrapped in a thunk.

Additionally the sender's entry on the sendq is now stale, as the operation is already complete; we try to pop it off the queue at the line indicated with *, but that could fail due to concurrent queue modification. In that case, no biggie, someone else will do the collect our garbage for us.

The pessimistic case is a bit more involved. It's the last bit of code though; almost done here! I express the pessimistic phase as a function of the operation's block function.

(define (pessimistic block)
  ;; For consistency with optimistic phase, result of
  ;; pessimistic phase is a thunk that "perform" will
  ;; apply.
  (lambda ()
    ;; 1. Suspend the thread.  Expect to be resumed
    ;; with a thunk, which we arrange to invoke directly.
       (lambda (k)
        (define (resume values-thunk)
          (schedule (lambda () (k values-thunk))))
        ;; 2. Make a fresh opstate.
        (define state (make-atomic-box 'W))
        ;; 3. Call op's block function.
        (block resume state))))))

So what about that state variable? Well basically, once we publish the fact that we're ready to perform an operation, fibers from other cores might concurrently try to complete our operation. We need for this perform invocation to complete at most once! So we introduce a state variable, the "opstate", held in an atomic box. It has three states:

  • W: "Waiting"; initial state

  • C: "Claimed"; temporary state

  • S: "Synched"; final state

There are four possible state transitions, of two kinds. Firstly there are the "local" transitions W->C, C->W, and C->S. These transitions may only ever occur as part of the "retry" phase a block function; notably, no remote fiber will cause these transitions on "our" state variable. Remote fibers can only make the W->S transition, committing an operation. The W->S transition can also be made locally of course.

Every time an operation is instantiated via the perform function, we make a new opstate. Operations themselves don't hold any state; only their instantiations do.

The need for the C state wasn't initially obvious to me, but after seeing the recv-op block function below, it will be clear to you I hope.

block functions

The block function itself has two jobs to do. Recall that it's called after the calling fiber was suspended, and is passed two arguments: a procedure that can be called to resume the fiber with some number of values, and the fresh opstate for this instantiation. The block function has two jobs: it needs to publish the resume function and the opstate to the channel's recvq, and then it needs to try again to receive. That's the "retry" phase I was mentioning before.

Retrying the recv can have three possible results:

  1. If the retry succeeds, we resume the sender. We also have to resume the calling fiber, as it has been suspended already. In general, whatever code manages to commit an operation has to resume any fibers that were waiting on it to complete.

  2. If the operation was already in the S state, that means some other party concurrently completed our operation on our behalf. In that case there's nothing to do; the other party resumed us already.

  3. Otherwise if the operation couldn't proceed, then when the other party or parties arrive, they will be responsible for completing the operation and ultimately resuming our fiber in the future.

With that long prelude out of the way, here's the gnarlies!

(define (block-recv ch resume-recv recv-state)
  (match ch
    (($ $channel recvq sendq)
     ;; Publish -- now others can resume us!
     (enqueue! recvq (vector resume-recv recv-state))
     ;; Try again to receive.
     (let retry ()
       (let ((q (atomic-ref sendq)))
         (match q
           ((head . tail)
            (match head
              (#(val resume-send send-state)
               (match (CAS! recv-state 'W 'C)   ; Claim txn.
                  (match (CAS! send-state 'W 'S)
                    ('W                         ; Case (1): yay!
                     (atomic-set! recv-state 'S)
                     (CAS! sendq q tail)        ; Maybe GC.
                     (resume-send (lambda () (values)))
                     (resume-recv (lambda () val)))
                    ('C                         ; Conflict; retry.
                     (atomic-set! recv-state 'W)
                    ('S                         ; GC and retry.
                     (atomic-set! recv-state 'W)
                     (CAS! sendq q tail)
                 ('S #f)))))                    ; Case (2): cool!
           (() #f)))))))                        ; Case (3): we wait.

As we said, first we publish, then we retry. If there is a sender on the queue, we will try to complete their operation, but before we do that we have to prevent other fibers from completing ours; that's the purpose of going into the C state. If we manage to commit the sender's operation, then we commit ours too, going from C to S; otherwise we roll back to W. If the sender itself was in C then we had a conflict, and we spin to retry. We also try to GC off any completed operations from the sendq via unchecked CAS. If there's no sender on the queue, we just wait.

And that's it for the code! Thank you for suffering through this all. I only left off a few details: the try function can loop if sender is in the C state, and the block function needs to avoid a (choice-op (send-op A v) (recv-op A)) from sending v to itself. But because opstates are fresh allocations, we can know if a sender is actually ourself by comparing its opstate to ours (with eq?).

what about select?

I started about all this "op" business because I needed to annotate the arguments to select. Did I actually get anywhere? Good news, everyone: it turns out that select doesn't have to be a primitive!

Firstly, note that the choose-op try function just needs to run all try functions of sub-operations (possibly in random order), returning early if one succeeds. Pretty straightforward. And actually the story with the block function is the same: we just run the sub-operation block functions, knowing that the operation will commit at most one time. The only complication is plumbing through the respective wrap functions to all of the sub-operations, but of course that's the point of the wrap facility, so we pay the cost willingly.

(define (choice-op . ops)
  (define (try)
      (($ $op sub-try sub-block sub-wrap)
       (define thunk (sub-try))
       (and thunk (compose sub-wrap thunk))))
  (define (block resume opstate)
      (($ $op sub-try sub-block sub-wrap)
       (define (wrapped-resume results-thunk)
         (resume (compose sub-wrap results-thunk)))
       (sub-block wrapped-resume opstate)))
  (define wrap values)
  (make-op try block wrap))

There are optimizations possible, for example to randomize the order of visiting the sub-operations for more less deterministic behavior, but this is really all there is.

concurrent ml is inevitable

As far as I understand things, the protocol to implement CML-style operations on channels in a lock-free environment are exactly the same as what's needed if you wrote out the recv function by hand, without abstracting it to a recv-op.

You still need the ability to park a fiber in the block function, and you still need to retry the operation after parking. Although try is just an optimization, it's an optimization that you'll want.

So given that the cost of parallel CML is necessary, you might as well get what you pay for and have your language expose the more expressive CML interface in addition to the more "standard" channel operations.

concurrent ml between pthreads and fibers

One really cool aspect about implementing CML is that the bit that suspends the current thread is isolated in the perform function. Of course if you're in a fiber, you suspend the current fiber as we have described above. But what if you're not? What if you want to use CML to communicate between POSIX threads? You can do that, just create a mutex/cond pair and pass a procedure that will signal the cond as the resume argument to the block function. It just works! The channels implementation doesn't need to know anything about pthreads, or even fibers for that matter.

In fact, you can actually use CML operations to communicate between fibers and full pthreads. This can be really useful if you need to run some truly blocking operation in a side pthread, but you want most of your program to be in fibers.

a meta-note for a meta-language

This implementation was based on the Parallel CML paper from Reppy et al, describing the protocol implemented in Manticore. Since then there's been a lot of development there; you should check out Manticore! I also hear that Reppy has a new version of his "Concurrent Programming in ML" book coming out soon (not sure though).

This work is in Fibers, a concurrency facility for Guile Scheme, built as a library. Check out the manual for full details. Relative to the Parallel CML paper, this work has a couple differences beyond the superficial operation/perform event/sync name change.

Most significantly, Reppy's CML operations have three phases: poll, do, and block. Fibers uses just two, as in a concurrent context it doesn't make sense to check-then-do. There is no do, only try :)

Additionally the Fibers channel implementation is lockless, with an atomic sendq and recvq. In contrast, Manticore uses a spinlock and hence needs to mask/unmask interrupts at times.

On the other hand, the Parallel CML paper included some model checking work, which Fibers doesn't have. It would be nice to have some more confidence on correctness!

but what about perf

Performance! Does it scale? Let's poke it. Here I'm going to try to isolate my tests to measure the overhead of communication of channels as implemented in terms of Parallel CML ops. I have more real benchmarks for Fibers on a web server workload where it does well, but here I am really trying to focus on CML.

My test system is a 2 x E5-2620v3, which is two sockets each having 6 2.6GHz cores, hyperthreads off, performance governor on all cores. This is a system we use for Snabb testing, so the first core on each socket handles interrupts and all others are reserved; Linux won't schedule anything on them. When you run a fibers system, it will spawn a thread per available core, then set the thread's affinity to that core. In these tests, I'll give benchmarks progressively more cores and see how they do with the workload.

So this is a benchmark measuring total message sends per second on a chain of fibers communicating over channels. For 0 links, that means that there's just a sender and a receiver and no intermediate links. For 10 links, each message is relayed 10 times, for 11 total sends in the chain and 12 total fibers. For 0 links we expect pretty much no parallel speedup, and no slowdown, and that's what we see; but when we get to more links, we should expect more throughput. The fibers are allocated to cores at random (a randomized round-robin initial scheduling, then after that fibers have core affinity; though there is a limited work-stealing phase).

You would think that the 1-core case would be the same for all of them. Unfortunately it seems that currently there is a fixed cost for bouncing through epoll to pick up new I/O tasks, even though there are no I/O runnables in this test and the timeout is 0, so it will return immediately. It's definitely something to look into as it's a cost that all cores are paying.

Initially I expected a linear speedup but that's not what we're seeing. But then I thought about it and revised my expectations :) As we add more cores, we add more communication; we should see sublinear speedups as we have to do more cross-core wakeups and synchronizations. After all, we aren't measuring a nice parallelizable computational workload: we're measuring overhead.

On the other hand, the diminishing returns effect is pretty bad, and then we hit the NUMA cliff: as we cross from 6 to 7 cores, we start talking to the other CPU socket and everything goes to shit.

But here it's hard to isolate the test from three external factors, whose impact I don't understand: firstly, that Fibers itself has a significant wakeup cost for remote schedulers. I haven't measured contention on scheduler inboxes, but I suspect one issue is that when a remote scheduler has decided it has no runnables, it will sleep in epoll; and to wake it up we need to write on a socketpair. Guile can avoid that when there are lots of runnables and we see the remote scheduler isn't sleeping, but it's not perfect.

Secondly, Guile is a bytecode VM. I measured that Guile retires about 0.4 billion instructions per second per core on the test machine, whereas a 4 IPC native program will retire about 10 billion. There's overhead at various points, some of which will go away with native compilation in Guile but some might not for a while, given that Go (for example) has baked-in support for channels. So to what extent is it the protocol and to what extent the implementation overhead? I don't know.

Finally, and perhaps most importantly, we can't isolate this test from the garbage collector. Guile still uses the Boehm GC, which is just OK I think. It does have a nice parallel mark phase, but it uses POSIX signals to pause program threads instead of having those threads reach safepoints; and it's completely NUMA-unaware.

So, with all of those caveats mentioned, let's see a couple more graphs :) Firstly, similar to the previous one, here's total message send rate for N pairs of fibers that ping-pong their message back and forth. Core allocation was randomized round-robin.

My conclusion here is that when more fibers are runnable per scheduler turn, the overhead of the epoll phase is less.

Here's a test where there's one fiber producer, and N fibers competing to consume the messages sent. Ultimately we expect that the rate will be limited on the producer side, but there's still a nice speedup.

Next is a pretty weak-sauce benchmark where we're computing diagonal lengths on an N-dimensional cube; the squares of the dimensions happen in parallel fibers, then one fiber collects those lengths, sums and makes a square root.

The workload on that one is just very low, and the serial components become a bottleneck quickly. I think I need to rework that test case.

Finally, there's a false sieve of Erastothenes, in which every time we find a prime, we add another fiber onto the sieve chain that filters out multiples of that prime.

Even though the workload is really small, we still see speedups, which is somewhat satisfying. Still, on all of these, the NUMA cliff is something fierce.

For me what these benchmarks show is that there are still some bottlenecks to work on. We do OK in the handful-of-cores scenario, but the system as a whole doesn't really scale past that. On more real benchmarks with bigger workloads and proportionally much less communication, I get much more satisfactory results; but those tend to be I/O heavy anyway, so the bottleneck is elsewhere.

closing notes

There are other parts to CML events, namely guard functions and withNack functions. My understanding is that these are implementable in terms of this "primitive" CML as described here; that was a result of earlier work by Matthew Fluet. I haven't actually implemented these yet! A to-do item, truly.

There are other event types in CML systems of course! Besides being able to implement operations yourself, there are built-in condition variables (cvars), timeouts, thread join events, and so on. The Fibers manual mentions some of these, but it's an open set.

Finally and perhaps most significantly, Aaron Turon did some work a few years ago on "Reagents", a pattern library for composing parallel and concurrent operations, initially in Scala. It's claimed that Reagents generalizes CML. Is this the case? I am looking forward to finding out.

OK, that's it for this verrrrry long post :) I hope that you found that this made parallel CML seem a bit more approachable and interesting, whether as a language implementor, a library implementor, or a user. Comments and corrections welcome. Check out Fibers and give it a go!

147 responses

  1. Adam Solove says:

    I've been waiting for this post for a while, but it was worth it! Thanks for the great walkthrough of your thought process and implementation decisions.

  2. solrize says:

    I wonder if you've looked at Haskell's Software Transactional Memory (STM). I think Clojure implements something similar. Simon Peyton Jones' paper "Composable memory transactions" is worth reading:

    This is important if you want to share mutable data between threads.

  3. Arne Babenhauserheide says:

    Thank you for your great writeup!

    If you want to look into STM, you can also get current information from the pypy devs, with some benchmarks:

    They try to get that funded:

    Here’s a talk about STM in pypy from 2 years ago: →

  4. William ML Leslie says:

    Thanks Andy, this is impressive work and a very neat implementation.

    Looking at the scalability graphs, I'm reminded that when we discussed the downsides to bi-directional synchronisation, I think we mostly talked engineering concerns. Once you start bringing NUMA into the picture, there is a definite performance cost. If you can release a value onto a buffered queue and a later thread can acquire it, both threads can get useful work done in the meantime.

    I've had a few experiences where moving away from bi-directional synchronisation has been good for performance, so I wonder if you got the chance to run the same benchmark in 8sync or so to compare the numbers?

  5. Jules Jacobs says:

    Reagents do generalise CML. The main difference is that CML only allows you to combine operations with select (op1 OR op2) while Reagents also allow you to combine operations into a transaction (op1 AND op2 for independent operations and op1 THEN op2 for dependent ones). Reagents are lightweight in that the library analyses the combined operation and then figures out an efficient CAS scheme to execute it. Reagents also include some more low level operations, such as CAS as a reagent.

  6. Write me an essay says:

    Vegetable strands are by and large in view of courses of action of cellulose, frequently with lignin cases.

  7. free crossword puzzles says:

    Here you can learn number of new words and add it into your knowledge as well and play the puzzle game online for free.

  8. Matt Wette says:

    Hi Andy, I have used SPIN for verifying concurrency protocols. Maybe of interest ...

  9. Hire Someone to Do Assignment says:

    Here is test code to print "hi, world" to the comfort. It brings forth a string which makes a channel for strings. This string at that point brings forth another string which prints the principal string that is gotten on the channel. The previous string at that point sends the "welcome, world\n" string on the channel.

  10. Matt Wette says:

    Hey again! So I am reading the P-C-ML paper now and going through the 1.0.0 code. My goal is to submit patches to get fibers working with kqueue (macOS and FreeBSD). Also, this stuff reminds me of rendezvous in Ada.

  11. says:

    It is one of the best concepts I have ever seen it should help people a lot. Also, this clears that whole miscalculation.

  12. says:

    I haven't read the last article so I have a bit confusion may be I should read the previous article.

  13. says:

    This explanation is good if a person completely understands this concept it will don't have any problems with it's working.

  14. WritingEssay says:

    The message that is given off in a channel send/get operation is never "claimed" by the channel; it is either possessed by a sender who is sitting tight at the meeting point for a recipient, or it's acknowledged by a collector. After the exchange is finished, the two gatherings proceed on.

  15. says:

    It appeared to be old and aged and possibly channels were similarly as expressive. I looked all the more profoundly into this issue and it appeared CML is more expressive than just channels yet damn, it looked confused to actualize.

  16. duct ac maintenance in dubai says:

    I haven't perused the last article so I have a bit perplexity might be I should read the past article.

  17. Best Essay Writing Service says:

    It is vital that we read blog put up very carefully. I am already carried out it and locate that this post is virtually great.

  18. says:

    There are additionally other (somewhat bizarre and particular) methods for doing this by utilizing the backchannel. These are custom usage with either sending an officially existing access token to the bot or utilizing home prepared enchantment number generators. Be that as it may,

  19. write my assignment says:

    Once you reach maturity, muscle fibers typically do not increase in number. There are "satellite cells" around each muscle fiber. The muscle fiber increases in size through "hypertrophy," which is enlargement by these satellite cells fusing with the existing fiber.

  20. Custom Essay Writing Service says:

    In the pessimistic phase, we park the calling fiber, put up the reality that we are equipped and watching for the operation, then to remedy the race condition we should strive once more to finish the operation.

  21. Online Essay Writing Services UK says:

    When having to travel on an urgent basis, with no time to concentrate on an elaborate and intricate essay. Then, students must see to it that they look into online essay writing services UK, as they most likely to relieve stress and pressure surmounted on the student.

  22. Dissertation Service says:

    In the event that a sender and a recipient simultaneously touch base at a channel, it may be the case that they simultaneously do the hopeful check, don't see that the other is there, at that point they both suspend, sitting tight for each other to arrive: stop.

  23. Essay Writing Service UK Cheap says:

    They may not need to worry about it due to the fact there's a large list of things which might be essential right now, that they also do not worry approximately.

  24. UK Best Essay says:

    In the skeptical stage, we stop the calling fiber, set up the truth that we are prepared and looking for the activity, at that point to cure the race condition we ought to endeavor afresh to complete the task.

  25. Affordable SEO Services For Small Business says:

    Following the normal so consistently can turn out to be a significant tumultuous errand. That is the reason you require somebody to help with composing errands. Get Professional Essays Writers UK. On the off chance that you don't know who to contact, call us at Professional Essays Writers immediately generally advantageous and most master help.

  26. Buy Dissertation Online UK says:

    This service caters them from topic suggestion to proofread services and their dedicated team is also willing to revise your academic papers if they are contrary to what you asked for.

  27. Order Essay Online UK says:

    Why would you be about the advertising essays when Best Assignment Writer consultants let you in expertise the standards and the later developments of marketing? Get in touch with them now, and you are in for a deal with of awesome grades truly.

  28. Walatra Sehat Sendi says:

    Why would you be about the advertising essays when Best Assignment Writer consultants let you in expertise the standards and the later developments of marketing? Get in touch with them now, and you are in for a deal with of awesome grades truly.

  29. google says:

    Why would you be about the advertising essays when Best Assignment Writer consultants let you in expertise the standards and the later developments of marketing? Get in touch with them now, and you are in for a deal with of awesome grades truly.

  30. James Taylor says:

    Essay putting together could possibly get exceedingly lazy , most importantly the moment they encompass engaging in cumbersome as well as limitless errands but nevertheless Cheap Assignment Help UK who stimulates you ends up almost all assignments before the due date , with the ambitions that you can begin to grasp your potential and do the job in your own realistic lifestyle . We permit understudy to be at ease by giving away these individuals professional and astonishing level of essayists' job that is certainly experienced by our clients. We furthermore offer immeasurable rechecking and modifying to improve your projects such as the approach you will need.

  31. british essay writers says:

    The selection always must be created nicely! I like that you've included the graphics in your article - now it's much easier to conceive.

  32. Custom Essay Writing Services UK says:

    For what reason would you be about the publicizing papers when Best Assignment Writer advisors given you access mastery the norms and the later advancements of advertising? Connect with them now, and you are in for an arrangement with of magnificent evaluations genuinely.

  33. Essay Writing Service says:

    All things considered, there has been a demonstrated world out there that is parallel to our own! In any case, it's not a similar idea of parallel that you read in sci-fi books. In principle, the parallel world to our reality is precisely the same, with several extraordinary.

  34. sonu says:

    This is really very goood for the everyone

  35. Law Dissertation Writing says:

    Thank you so much for your great data sharing.

  36. STATA Assignment Help says:

    My Friend Recommended This Blog And He Was Totally Right Keep Up The Fantastic Work!

  37. STATA Assignment Help says:

    Good Way Of Telling, Good Post To Take Facts Regarding My Presentation Subject Matter, Which I Am Going To Deliver In My College

  38. torrents says:

    Thanks for informative info and detail instruction. It's really useful for me

  39. cheap essay writing service says:

    It decreases and improves coordinations: Each strategy for preparing has its advantages and disadvantages. Eye to eye preparing when driven by an expert mentor can accomplish significant outcomes; notwithstanding, dealing with the coordinations of the considerable number of members can be a bad dream. However, when the live projects are diminished and the preparation is upheld by video meetings and elearning, at that point the effect isn't imperiled. The last two preparing techniques diminishes the requirement for coordinations and is additionally eco-accommodating; e-duplicates of the preparation materials can be share in this manner decreasing the requirement for printing.

  40. Help With Dissertation UK says:

    It is dependent upon you how you need to finish your training when you take an online class.

  41. Best Assignment Writing Services says:

    Sometimes the definition of records parallelism is loosened from the training degree to the subroutine or maybe the program level, although that is extra as it should be referred to as SPMD shared application, more than one statistics. Clusters that use message-passing among servers might run the identical program on special statistics, however branching and other manipulate differences suggest they could now not necessarily comply with the same education collection and could take distinct amounts of time to complete, ensuing in load imbalance.

  42. help with essay says:

    At long last, changes make your written work sound more expert and less like talked dialect. Enhancing Your Essay: Choosing the Perfect Words Enhancing Your Essay: Choosing the Perfect Words Step by step instructions to Use Sentence Starter Lists The most imperative tip for enhancing your written work rapidly is to tail one run the show:

  43. java collections interview questions says:

    java collections interview questions. In Java Collections are the group of objects represents a single entity. A collection is a java object that is used to group both homogeneous and heterogeneous elements.

  44. Dissertation help uk says:

    Difficult to state, yet yes, I have a tendency to concur with you. I for one am not on Facebook, nor do I claim an advanced mobile phone for the plain reasons most technologists have considered, however couldn't manage without.

  45. Custom Dissertation Help says:

    Hot water and soap to wash dishes like Dawn. You will have to do several washes and rinses. You can put it in a lingerie bag and go through the spin cycle to extract as much water, but do not let it shake in the washing machine as you will feel.

  46. Dissertation Experts says:

    What if you get expert options for your essays or dissertation without any charges. Yes off course Dissertation help is offering FREE consultation for dissertations. Get it Now!

  47. Tips for write college admisison essay says:

    In this article author shared his experience in fiber.

  48. assignment help uk says:

    I've been looking ahead to this post for a minute, however, it absolutely was pricing it! Thanks for the good walkthrough of your thought method and implementation selections.

  49. RichSEO - SEO in Coimbatore says:

    A good website for knowing fiber


    Thanks for sharing article on fiber.


    Light-weight fiber applications need to briefed for better understanding.alishabodymassage


    Fiber strength tester details needed.

    super article

  53. MIPPIN MOBILES says:

    Nice article on fibers.

  54. openload stream authorization says:

    Nice explanation, you have explained briefly on fiber. Keep sharing more.

  55. Anwar Ali says:

    We do academic and content writing with no plagiarism from our highly professional academic writers for different niches
    Academic writing, thesis, assignment, research paper, content.
    Academic writing from professional to get you successful in your exams

  56. shark il primo squalo streaming says:

    Date Night Movies: I am a big proponent of continuing to date your spouse after marriage. For the unmarried – lots of time together is a good thing too.

  57. happy wheels says:

    Using a RPC channel is great and helps me a lot.

  58. wordpress homepage says:

    I explain the Top 5 Reasons To use Wordpress for Your Blog. I break down why Google love WordPress and the basic features to make your blogging even better.

  59. twitch says:

    wow! These look perfect!! this is truly an amazing article

  60. how to download music online says:

    This is an awesome blog. You provide very useful data.

  61. preschool photography agency says:

    Thanks for sharing,

  62. email marketing says:

    Great Article,

  63. Assignment Help says:

    You should have clearly define topics which you want to learn and start collecting study material according to that topics. Also some assignment help service you can use to get knowledge as well. Just proceed accordingly and don’t waste your time because time is precious.

  64. Assignment Help says:

    A high-standard post with all imperative information about Assignment Help UK services. Looking forward to availing the premium services.

  65. Best law essay writing service says:

    Writing a Law essay is not an easy task that anyone can write it by their own, therefore many students hire our services for their essay writing to get professional and unique content.

  66. rina kha says:

    Thai Dry Massage is a dry massage with no oil. Thai massage is also called Thai yoga massage, because the therapist uses his or her hands, knees, legs, and feet to move you into a series of yoga­like stretches.

  67. Essay Writing Service uk says:

    Professional essay writers in UK offers custom essay, marketing essay, law essay, management essay, economic essay writing service with reasonable price.

  68. Best UK Essay Writing Service says:

    I've been sitting tight for this post for some time, however it was justified, despite all the trouble! Much obliged for the incredible walkthrough of your manner of thinking and usage choices.

  69. British Dissertation Writers says:

    As exams approach the level of stress and anxiety tends to increase. Hence in order to avoid this sense of disorientation and confusion students should make use of an essay writing service which can take care of your workload while you focus on your exam preparation.

  70. eyelash extensions london says:

    The benefits of eyelash extensions is a long and luscious lashes, this is what almost every woman wants to have: longer, fuller and fluttery eyelashes.
    But many of London`s women struggle with finding the correct lash mascara and find extremely difficult to wear strip lashes and the solution is eyelash extensions.

  71. Eyelash Extensions says:

    If you’d have asked me a couple of months in the past if I fancied spending two plus a half hrs possessing eyelash extensions glued to my own eyelashes, I’d have given you a company slap across your proper cheek.

  72. Assignment Expert says:

    Only Professional Writers Can Make This Kind Of Material, Cheers

  73. Buy Assignment says:

    John Arnold Is An Academic Writer Of The Dissertation-Guidance. Who Writes Quality Academic Papers For Students To Help Them In Accomplishing Their Goals.

  74. individual eyelashes says:

    There endless possibilities and styles of lashes, which vary from classic lashes to russian volume lashes. If you want to get a free lash consultation, and see which style is best for your…

  75. lash extensions in london says:

    Individual lashes are one of the various types of lash extension applied to one natural lash only, they must be applied by a qualified lash stylist, check this article and learn more about eyelash extensions.

  76. Tony Robbins says:

    Have you attended tony robbins in london. If you are looking for Tony Robbins in London, than you just came to the right place! Here you will find all info about UPW London. Check this!

  77. James Jasper says:

    This is James Jasper from the UK. He is an expert writer in the field of law dissertation help and working for British Coursework Help. All UK students are getting high-grade coursework papers at the lowest price and achieving success. His aim is to satisfy student writing and writing.

  78. Lisa says:

    How is your doings today? I hope it felt so right.

  79. Agnes dos Santos says:

    However every one of the frustration signifies salons could be misleading in what they can be giving which results in further confusion and quite often gives lash extension a poor brand. eyelash extensions near me

  80. Best Assignment Writing Service UK says:

    It decreases and improves coordinations: Each strategy for getting ready has its advantages and disadvantages. Eye to eye making ready while pushed through an professional mentor can accomplish giant outcomes; notwithstanding, handling the coordinations of the large number of participants may be a horrific dream. However, while the stay projects are dwindled and the coaching is upheld with the aid of video conferences and elearning, at that factor the impact isn't imperiled. The ultimate two making ready strategies diminishes the requirement for coordinations and is moreover eco-accommodating; e-duplicates of the education materials can be proportion in this way decreasing the requirement for printing.

  81. ROSE SOLIZ says:


  82. alex juvion says:

    The ultimate two making ready strategies diminishes the requirement for coordinations and is moreover eco-accommodating; e-duplicates of the education materials can be proportion in this way decreasing the requirement for printing.

  83. raveena kapoor, says:

    thanks for sharing info, If you’d have asked me a couple of months in the past if I fancied spending two plus a half hrs possessing eyelash extensions glued to my own eyelashes, have given you a company slap across your proper cheek.

  84. shark fishing games says:

    For me what these benchmarks show is that there are still some bottlenecks to work on.

  85. essay help online says:

    Thank you so much for sharing this amazing and useful information with us, you have got an interesting way of communicating the message in your blogs, really liked it, keep bogging and keep sharing the useful work.

  86. legge di attrazione says:

    85 comment and none of them is about this topic? What's that?

  87. lei da Atração says:

    I read one article on this topic and I found out that concurrent ML is a coding language in Joomla.

  88. tobyy warner says:

    Interesting article! Thank you for sharing them! I hope you will continue to have similar posts to share with everyonedissertation writer uk

  89. says:

    Thanks for sharing interesting article.

  90. Jack Katler says:

    Wonderful Article.
    Check out our article on Five most common challenges International students face while studying and how to overcome these challenges

  91. UK Essay Writing Service says:

    That’s something out of the box, really appreciated, this is what I had been looking for some weeks. Very Nice and informative article. Furthermore, you have got an interesting way of communicating your message. Thanks Mate! Keep sharing this useful blogs. Cheers!!

  92. says:

    thank You!

  93. Dissertation Help says:

    We are world's number 1 leading company who provide online thesis paper, research paper n guide, proposal writing and Dissertation Help UK. Your academic future is in safe hands.

  94. Tealand says:

    Yes I agree with you in this regard.

  95. Krazy Dog says:

    Visit this to read about pets including dogs

  96. custom furniture upholstery arizona says:

    Sometimes the description of records parallelism is untied from the exercise degree to the subroutine or maybe the platform level.

  97. Wifi Hire says:

    Thanks for useful info and detail lessons. It's really valuable to me.

  98. Seo Services says:

    very helpful for me thanks

  99. Assignment Help says:

    Nice Post
    My name is Joseph Wong and thanks for sharing this information.
    Get the best Assignment Help in Australia 24*7.

  100. Write My Essays says:

    Are you looking for a professional and qualified essay writing service that can provide work right on the set deadline? Then, make it a point to make use of the foremost literature review writing service, which is write my essays. This service has cemented this customer base, as they have benchmarked stellar standards, they don’t compromise or falter in any which way and they remain steadfast throughout the provision of their services. Also, they provide their customers with a essay writing service, in order to allow them to make a reasoned and logical decision, rather than merely choosing a service.

  101. Locus rags says:

    Nice Post
    My name is Joseph Wong and thanks for sharing this information.
    Best Web and Mobile Application solution provider company Locus Rags.

  102. Top Difference Between says:

    Best article to a new concurrent of channels and cores but it's really important to make sure the latest things and technology which are coming rapidly in world.

  103. PTV Sports HD says:

    Best article thanks for sharing.

  104. Tony Hark says:

    It's one of interesting topic of concurrent in electronics and its amazing to get know about the importance of education in Electronic field.

  105. Write my assignment for me says:

    Best article to another simultaneous of channels and centres however it's extremely vital to ensure the most recent things and innovation which are coming quickly in the world.

  106. telstra webmail login says:

    Hi, I found your website by the utilisation of Google even as hunting down a comparable subject, your site came up, it seems very good. I've bookmarked it in my google bookmarks.

  107. Essay writing services says:

    Excellent and nice post. It will beneficial for everyone. Thanks for sharing such a wonderful post. Avail No 1 Essay writing services UK from certified PhD writers. It is extremely helpful for me. You can email us at or Phone Number - 020 8144 9988

  108. dissertation writing services says:

    thanks for sharing the article regarding the topic.....

  109. UK Essay Writing says:

    Best article and a really amazing blog post just love it !! Thanks for sharing will subscribe this blog for future use.

  110. active target promo codes says:

    Sometimes the description of records parallelism is untied from the exercise degree to the subroutine or maybe the platform level.

  111. CallTutors says:

    I loved the article, keep updating interesting articles. I will be a regular reader… I am offering assignment help to students over the globe at a low price.

  112. CodeAvail says:

    I found this one pretty fascinating and it should go into my collection. Very good work! I am Impressed. We appreciate that please keep going to write more content. We are the assignment helper, we provide services all over the globe. We are best in these:- services

  113. OwnlyDigital says:

    Thanks for sharing this information. I have shared this link with other keep posting such information to provide best in class law assignment help online at very affordable prices.

  114. statanalytia says:

    Many of the people are depressed about the problems of essay writing. Well, don’t worry about that because we are providing this service at a very reasonable price.

  115. Thesis Writing Service says:

    I find this article useful. Kindly share more such articles so that I can get better insight.

  116. Assignment Writing Services says:

    Looking great work dear, I really appreciated to you on this quality work. Nice post!! these tips may help me in the future.

  117. essay writing company says:

    If you are waiting for an opportunity in-between your web content to do something that you are eagerly want to do, then give up. The perfect time will never come for you; you will have to create it yourself. Get Essay Writing Service UK from Law Essay Writers, and we will give you said opportunity.

  118. avengers 4 scarlett johansson vest says:

    I am really very impress to see this post. I am going to bookmark this post. You are doing very well job.

  119. Best Coursework says:

    Let me take a nap... great shot, anyway. My 20 year old son rates this illustration very fabulous, friend.

  120. free essays to copy says:

    During higher studies in colleges, students often have to prepare multiple documents, quizzes, and surprise tests. This is the main reason why most students search for MyAssignmentHelp> over the internet and choose only the most proficient and trusted academic writing experts for 100% free plagiarism free essays.

  121. Victoria says:

    Useful post. Capture full screen or customized screen along with audio by using best screen recorder for Windows.

  122. temple run says:

    The information you share is very good and useful to me and many others. It is closely related to my work and helped me grow.

  123. Case study writing services says:

    Students Assignment Help is the best case study writing services provider agency which helps students to accomplish assignment writing task easily. We have3 000+ expert assignment writers from top colleges and universities around the world.

  124. reviews says:

    nice post!

  125. Body massage in Bangalore says:

    We provide the best body to body massage service in Bangalore.

  126. dissertation writing services uk says:

    Thanks for your excellent posting. I really enjoyed reading this blog.

  127. Vadodara escorts services says:

    Your assignment is really awesome and i really enjoy it to read.. thanks

  128. data mining services says:

    Thanks for posting this info. Kindly share more such articles so that I can get better insight.

  129. cheap essay writing service uk says:

    I really thank you for this post. For a hassle-free experience, make sure to give our cheap essay writing service UK a try. We guarantee that you won’t be disappointed. Register today!

  130. Informative Speech Essays says:

    Our Pay for informative speech essays online customer access our dissertation services knowing that they are assured of the best quality essay lest they get their money back.

  131. Nursing Papers Made Easy says:

    All our Nursing Papers Made Easy that we provide to the client are assessed for high quality and any other issues that could affect the client’s grade.

  132. star tv canlı says:

    Turkiye'nin ilk ozel televizyon kanalı 1989 yilinda kurulmustur. Dogus Yayın Grubu uzerinden yayın yapmakta olan kanal star tv canlı ismi ile gunumuzde yayınlanmaktadir.Haber bultenleri, cesitli diziler, filmler ve yarisma programlarina star tv izle ekranlarindan ulasabilirsiniz.

  133. Cheap Nursing Term Papers says:

    We offer urgent essay writing service that meets all the instructions that the client provided and observe the academic writing standards that students should follow.

  134. Do My Computer Science Homework says:

    They Are Not Able To Finish The Writing Assignments On Time. For Some Students, Writing Any Writing Assignments Is Able To Waste Their Time

  135. essay writer review says:

    I am looking for how selectors review admission essay to write my complete admission essay.

  136. candy crush soda says:

    a very detailed and meticulous lesson, it really has a lot of values, I will learn a lot thanks

  137. Asbestos Removal Winnipeg says:

    Thanks to share this best information with us. Keep in touch with us to get more about best home repair service.

  138. says:

    Yolo Anonymous Messaging app is a social networking communication app developed by popshow inc. yolo Messaging app is the anonymous app for iOS devices.YOLO is a Snapchat-connected app that has detonated onto the online networking scene.

  139. carpet steam cleaner says:

    Simply, admirable what you have done here.

  140. Beautiful Place in Italy says:

    Amazing blog post, waiting for the next great article thanks for sharing.

  141. Lottery Sambad Today Result says:

    Nice post thanks for sharing simply amazing.

  142. Study helper says:

    MyAssignmentHelp is offering study helper or students studying in any degree or any college/university across the globe. We have an in-house team of PhD writers holding a vast experience in managing all study help.

  143. says:

    Very informative article. Thanks for posting

  144. Agence seo Tunisie says:

    Hi, Excellent tutorial article.

  145. vex 3 says:

    Thank you for writing! It is very easy to understand and detailed. I find it interesting, I hope you continue to have such good articles.

  146. Faux Leather Jackets says:

    The article you have shared is amazing and very informative. I really appreciate the post and content is great. Thanks!

  147. says:

    Writing an essay about communication is also one way of communicating your ideas to the readers. And to deliver thoughts in your communication essay ...

Leave a Reply