wingolog

growing fibers

27 June 2017 10:17 AM (guile | scheme | gnu | igalia | compilers | delimited continuations | prompts | fibers | concurrency)

Good day, Schemers!

Over the last 12 to 18 months, as we were preparing for the Guile 2.2 release, I was growing increasingly dissatisfied at not having a good concurrency story in Guile.

I wanted to be able to spawn a million threads on a core, to support highly-concurrent I/O servers, and Guile's POSIX threads are just not the answer. I needed something different, and this article is about the search for and the implementation of that thing.

on pthreads

It's worth being specific why POSIX threads are not a great abstraction. One is that they don't compose: two pieces of code that use mutexes won't necessarily compose together. A correct component A that takes locks might call a correct component B that takes locks, and the other way around, and if both happen concurrently you get the classic deadly-embrace deadlock.

POSIX threads are also terribly low-level. Asking someone to build a system with mutexes and cond vars is like building a house with exploding toothpicks.

I want to program network services in a straightforward way, and POSIX threads don't help me here either. I'd like to spawn a million "threads" (scare-quotes!), one for each client, each one just just looping reading a request, computing and writing the response, and so on. POSIX threads aren't the concrete implementation of this abstraction though, as in most systems you can't have more than a few thousand of them.

Finally as a Guile maintainer I have a duty to tell people the good ways to make their programs, but I can't in good conscience recommend POSIX threads to anyone. If someone is a responsible programmer, then yes we can discuss details of POSIX threads. But for a new Schemer? Never. Recommending POSIX threads is malpractice.

on scheme

In Scheme we claim to be minimalists. Whether we actually are that or not is another story, but it's true that we have a culture of trying to grow expressive systems from minimal primitives.

It's sometimes claimed that in Scheme, we don't need threads because we have call-with-current-continuation, an ultrapowerful primitive that lets us implement any kind of control structure we want. (The name screams for an abbreviation, so the alias call/cc is blessed; minimalism is whatever we say it is, right?) Unfortunately it turned out that while call/cc can implement any control abstraction, it can't implement any two. Abstractions built on call/cc don't compose!

Fortunately, there is a way to build powerful control abstractions that do compose. This article covers the first half of composing a concurrency facility out of a set of more basic primitives.

Just to be concrete, I have to start with a simple implementation of an event loop. We're going to build on it later, but for now, here we go:

(define (run sched)
  (match sched
    (($ $sched inbox i/o)
     (define (dequeue-tasks)
       (append (dequeue-all! inbox)
               (poll-for-tasks i/o)))
     (let lp ()
       (for-each (lambda (task) (task))
                 (dequeue-tasks))
       (lp)))))

This is a scheduler that is a record with two fields, inbox and i/o.

The inbox holds a queue of pending tasks, as thunks (procedure of no arguments). When something wants to enqueue a task, it posts a thunk to the inbox.

On the other hand, when a task needs to wait in some external input or output being available, it will register an event with i/o. Typically i/o will be a simple combination of an epollfd and a mapping of tasks to enqueue when a file descriptor becomes readable or writable. poll-for-tasks does the underlying epoll_wait call that pulls new I/O events from the kernel.

There are some details I'm leaving out, like when to have epoll_wait return directly, and when to have it wait for some time, and how to wake it up if it's sleeping while a task is posted to the scheduler's inbox, but ultimately this is the core of an event loop.

a long digression

Now you might think that I'm getting a little far afield from what my goal was, which was threads or fibers or something. But that's OK, let's go a little farther and talk about "prompts". The term "prompt" comes from the experience you get when you work on the command-line:

/home/wingo% ./prog

I don't know about you all, but I have the feeling that the /home/wingo% has a kind of solid reality, that my screen is not just an array of characters but there is a left-hand-side that belongs to the system, and a right-hand-side that's mine. The two parts are delimited by a prompt. Well prompts in Scheme allow you to provide this abstraction within your program: you can establish a program part that's a "system" facility, for whatever definition of "system" suits your purposes, and a part that's for the "user".

In a way, prompts generalize a pattern of system/user division that has special facilities in other programming languages, such as a try/catch block.

try {
  foo();
} catch (e) {
  bar();
}

Here again I put the "user" code in italics. Some other examples of control flow patterns that prompts generalize would be early exit of a subcomputation, coroutines, and nondeterminitic choice like SICP's amb operator. Coroutines is obviously where I'm headed here in the context of this article, but still there are some details to go over.

To make a prompt in Guile, you can use the % operator, which is pronounced "prompt":

(use-modules (ice-9 control))

(% expr
   (lambda (k . args) #f))

The name for this operator comes from Dorai Sitaram's 1993 paper, Handling Control; it's actually a pun on the tcsh prompt, if you must know. Anyway the basic idea in this example is that we run expr, but if it aborts we run the lambda handler instead, which just returns #f.

Really % is just syntactic sugar for call-with-prompt though. The previous example desugars to something like this:

(let ((tag (make-prompt-tag)))
  (call-with-prompt tag
    ;; Body:
    (lambda () expr)
    ;; Escape handler:
    (lambda (k . args) #f)))

(It's not quite the same; % uses a "default prompt tag". This is just a detail though.)

You see here that call-with-prompt is really the primitive. It will call the body thunk, but if an abort occurs within the body to the given prompt tag, then the body aborts and the handler is run instead.

So if you want to define a primitive that runs a function but allows early exit, we can do that:

(define-module (my-module)
  #:export (with-return))

(define-syntax-rule (with-return return body ...)
  (let ((t (make-prompt-tag)))
    (define (return . args)
      (apply abort-to-prompt t args))
    (call-with-prompt t
      (lambda () body ...)
      (lambda (k . rvals)
        (apply values rvals)))))

Here we define a module with a little with-return macro. We can use it like this:

(use-modules (my-module))

(with-return return
  (+ 3 (return 42)))
;; => 42

As you can see, calling return within the body will abort the computation and cause the with-return expression to evaluate to the arguments passed to return.

But what's up with the handler? Let's look again at the form of the call-with-prompt invocations we've been making.

(let ((tag (make-prompt-tag)))
  (call-with-prompt tag
    (lambda () ...)
    (lambda (k . args) ...)))

With the with-return macro, the handler took a first k argument, threw it away, and returned the remaining values. But the first argument to the handler is pretty cool: it is the continuation of the computation that was aborted, delimited by the prompt: meaning, it's the part of the computation between the abort-to-prompt and the call-with-prompt, packaged as a function that you can call.

If you call the k, the delimited continuation, you reinstate it:

(define (f)
  (define tag (make-prompt-tag))
  (call-with-prompt tag
   (lambda ()
     (+ 3
        (abort-to-prompt tag)))
   (lambda (k) k)))

(let ((k (f)))
  (k 1))
;; =& 4

Here, the abort-to-prompt invocation behaved simply like a "suspend" operation, returning the suspended computation k. Calling that continuation resumes it, supplying the value 1 to the saved continuation (+ 3 []), resulting in 4.

Basically, when a delimited continuation suspends, the first argument to the handler is a function that can resume the continuation.

tasks to fibers

And with that, we just built coroutines in terms of delimited continuations. We can turn our scheduler inside-out, giving the illusion that each task runs in its own isolated fiber.

(define tag (make-prompt-tag))

(define (call/susp thunk)
  (define (handler k on-suspend) (on-suspend k))
  (call-with-prompt tag thunk handler))

(define (suspend on-suspend)
  (abort-to-prompt tag on-suspend))

(define (schedule thunk)
  (match (current-scheduler)
    (($ $sched inbox i/o)
     (enqueue! inbox (lambda () (call/susp thunk))))))

So! Here we have a system that can run a thunk in a scheduler. Fine. No big deal. But if the thunk calls suspend, then it causes an abort back to a prompt. suspend takes a procedure as an argument, the on-suspend procedure, which will be called with one argument: the suspended continuation of the thunk. We've layered coroutines on top of the event loop.

Guile's virtual machine is a normal register virtual machine with a stack composed of function frames. It's not necessary to do full CPS conversion to implement delimited control, but if you don't, then your virtual machine needs primitive support for call-with-prompt, as Guile's VM does. In Guile then, a suspended continuation is an object composed of the slice of the stack captured between the prompt and the abort, and also the slice of the dynamic stack. (Guile keeps a parallel stack for dynamic bindings. Perhaps we should unify these; dunno.) This object is wrapped in a little procedure that uses VM primitives to push those stack frames back on, and continue.

I say all this just to give you a mental idea of what it costs to suspend a fiber. It will allocate storage proportional to the stack depth between the prompt and the abort. Usually this is a few dozen words, if there are 5 or 10 frames on the stack in the fiber.

We've gone from prompts to coroutines, and from here to fibers there's just a little farther to go. First, note that spawning a new fiber is simply scheduling a thunk:

(define (spawn-fiber thunk)
  (schedule thunk))

Many threading libraries provide a "yield" primitive, which simply suspends the current thread, allowing others to run. We can do this for fibers directly:

(define (yield)
  (suspend schedule))

Note that the on-suspend procedure here is just schedule, which re-schedules the continuation (but presumably at the back of the queue).

Similarly if we are reading on a non-blocking file descriptor and detect that we need more input before we can continue, but none is available, we can suspend and arrange for the epollfd to resume us later:

(define (wait-for-readable fd)
  (suspend
   (lambda (k)
     (match (current-scheduler)
       (($ $sched inbox i/o)
        (add-read-fd! i/o fd
                      (lambda () (schedule k))))))))

In Guile you can arrange to install this function as the "current read waiter", causing it to run whenever a port would block. The details are a little gnarly currently; see the Non-blocking I/O manual page for more.

Anyway the cool thing is that I can run any thunk within a spawn-fiber, without modification, and it will run as if in a new thread of some sort.

solid abstractions?

I admit that although I am very happy with Emacs, I never really took to using the shell from within Emacs. I always have a terminal open with a bunch of tabs. I think the reason for that is that I never quite understood why I could move the cursor over the bash prompt, or into previous expressions or results; it seemed like I was waking up groggily from some kind of dream where nothing was real. I like the terminal, where the only bit that's "mine" is the current command. All the rest is immutable text in the scrollback.

Similarly when you make a UI, you want to design things so that people perceive the screen as being composed of buttons and so on, not just lines. In essence you trick the user, a willing user who is ready to be tricked, into seeing buttons and text and not just weird pixels.

In the same way, with fibers we want to provide the illusion that fibers actually exist. To solidify this illusion, we're still missing a few elements.

One point relates to error handling. As it is, if an error happens in a fiber and the fiber doesn't handle it, the exception propagates out of the fiber, through the scheduler, and might cause the whole program to error out. So we need to wrap fibers in a catch-all.

(define (spawn-fiber thunk)
  (schedule
   (lambda ()
     (catch #t thunk
       (lambda (key . args)
         (print-exception (current-error-port) #f key args))))))

Well, OK. Exceptions won't propagate out of fibers, yay. In fact in Guile we add another catch inside the print-exception, in case the print-exception throws an exception... Anyway. Cool.

Another point relates to fiber-local variables. In an operating system, each process has a number of variables that are local to it, notably in UNIX we have the umask, the current effective user, the current directory, the open files and what file descriptors they are associated with, and so on. In Scheme we have similar facilities in the form of parameters.

Now the usual way that parameters are used is to bind a new value within the extent of some call:

(define (with-output-to-string thunk)
  (let ((p (open-output-string)))
    (parameterize ((current-output-port p))
      (thunk))
    (get-output-string p)))

Here the parameterize invocation established p as the current output port during the call to thunk. Parameters already compose quite well with prompts; Guile, like Racket, implements the protocol described by Kiselyov, Shan, and Sabry in their Delimited Dynamic Binding paper (well worth a read!).

The one missing piece is that parameters in Scheme are mutable (by default). Normally if you call (current-input-port), you just get the current value of the current input port parameter. But if you pass an argument, like (current-input-port p), then you actually set the current input port to that new value. This value will be in place until we leave some parameterize invocation that parameterizes the current input port.

The problem here is that it could be that there's an interesting parameter which some piece of Scheme code will want to just mutate, so that all further Scheme code will use the new value. This is fine if you have no concurrency: there's just one thing running. But when you have many fibers, you want to avoid mutations in one fiber from affecting others. You want some isolation with regards to parameters. In Guile, we do this with the with-dynamic-state facility, which isolates changes to the dynamic state (parameters and so on) within the extent of the with-dynamic-state call.

(define (spawn-fiber thunk)
  (let ((state (current-dynamic-state)))
    (schedule
     (lambda ()
       (catch #t
         (lambda ()
           (with-dynamic-state state thunk))
         (lambda (key . args)
           (print-exception (current-error-port) #f key args))))))

Interestingly, with-dynamic-state solves another problem as well. You would like for newly spawned fibers to inherit the parameters from the point at which they were spawned.

(parameterize ((current-output-port p))
  (spawn-fiber
   ;; New fiber should inherit current-output-port
   ;; binding as "p"
   (lambda () ...)))

Capturing the (current-dynamic-state) outside the thunk does this for us.

When I made this change in Guile, making sure that with-dynamic-state did not impose a continuation barrier, I ran into a problem. In Guile we implemented exceptions in terms of delimited continuations and dynamic binding. The current stack of exception handlers was a list, and each element included the exceptions handled by that handler, and what prompt to which to abort before running the exception handler. See where the problem is? If we ship this exception handler stack over to a new fiber, then an exception propagating out of the new fiber would be looking up handlers from another fiber, for prompts that probably aren't even on the stack any more.

The problem here is that if you store a heap-allocated stack of current exception handlers in a dynamic variable, and that dynamic variable is captured somehow (say, by a delimited continuation), then you capture the whole stack of handlers, not (in the case of delimited continuations) the delimited set of handlers that were active within the prompt. To fix this, we had to change Guile's exceptions to instead make catch just rebind the exception handler parameter to hold the handler installed by the catch. If Guile needs to walk the chain of exception handlers, we introduced a new primitive fluid-ref* to do so, building the chain from the current stack of parameterizations instead of some representation of that stack on the heap. It's O(n), but life is that way sometimes. This way also, delimited continuations capture the right set of exception handlers.

Finally, Guile also supports asynchronous interrupts. We can arrange to interrupt a Guile process (or POSIX thread) every so often, as measured in wall-clock or process time. It used to be that interrupt handlers caused a continuation barrier, but this is no longer the case, so now we can add pre-emption to a fibers using interrupts.

summary and reflections

In Guile we were able to create a solid-seeming abstraction for fibers by composing other basic building blocks from the Scheme toolkit. Guile users can take an abstraction that's implemented in terms of an event loop (any event loop) and layer fibers on top in a way that feels "real". We were able to do this because we have prompts (delimited continuation) and parameters (dynamic binding), and we were able to compose the two. Actually getting it all to work required fixing a few bugs.

In Fibers, we just use delimited continuations to implement coroutines, and then our fibers are coroutines. If we had coroutines as a primitive, that would work just as well. As it is, each suspension of a fiber will allocate a new continuation. Perhaps this is unimportant, given the average continuation size, but it would be comforting in a way to be able to re-use the allocation from the previous suspension (if any). Other languages with coroutine primitives might have an advantage here, though delimited dynamic binding is still relatively uncommon.

Another point is that because we use prompts to suspend fiberss, we effectively are always unwinding and rewinding the dynamic state. In practice this should be transparent to the user and the implementor should make this transparent from a performance perspective, with the exception of dynamic-wind. Basically any fiber suspension will be run the "out" guard of any enclosing dynamic-wind, and resumption will run the "in" guard. In practice we find that we defer "finalization" issues to with-throw-handler / catch, which unlike dynamic-wind don't run on every entry or exit of a dynamic extent and rather just run on exceptional exits. We will see over time if this situation is acceptable. It's certainly another nail in the coffin of dynamic-wind though.

This article started with pthreads malaise, and although we've solved the problem of having a million fibers, we haven't solved the communications problem. How should fibers communicate with each other? This is the topic for my next article. Until then, happy hacking :)

104 responses

  1. Alex @ Asset plus says:

    Were you able to create 1 million threads concurrently or in a queue-like fashion? Can you share resources between threads without getting into a deadlock situation?

  2. wingo says:

    I can create 1M threads concurrently, yes.

    As far as communication between threads goes, that's a topic for the next article :)

  3. Bengan says:

    Another great post! Most of it is way above my head, but I am really hyped about fibers, and I am looking forward to using it in the future!

  4. papas sushiria says:

    there is nothing entertaining website like a papa sushiria. I am a big game lover and i always choose games which belongs with the real world and i know very well papas cupCakeria is only a single website which serve genuine real games.

  5. assignment services says:

    Though you have tried amazingly but still I'm not satisfied with the derivation of 'Growing Fibers', there are some loop holes which need to be fixed..

  6. assignment services says:

    Though you have tried amazingly but still I'm not satisfied with the derivation of 'Growing Fibers', there are some loop holes which need to be fixed..

  7. custom essays uk says:

    Usually fibers aren't grow that quickly and after watching your derivation I simply don't understand that what was the purpose of that derivation...

  8. write my assignment says:

    Not really sure if it can also work for me but i believe in trying again and again.Good luck!

  9. james says:

    I like http://www.msh.org/blog/ website.

  10. assignment writing says:

    Itss worth being specific why POSIX threads are not a great abstraction. One is that they don't compose: two pieces of code that use mutexes won't necessarily compose together.

  11. Online Dissertation Help UK says:

    A to a great degree helpful book, with fascinating and significant data to what I require, and in addition "glib" data all over to make this a simple and charming read regardless of the specialized learning it contains

  12. do my essay for me uk says:

    IMO Guile is an attractive language! And Andy Wingo does an excellent job of describing it!

  13. Homepage says:

    money can mask the trauma of domestic violence. The second is to enjoy the trouble caused by gambling. When the spiritual control Dafa reveals itself

  14. Amanda Handerson says:

    woww that is really good

  15. Grace says:

    Good morning everybody! I am right here to notify you that the review states really reality. The publisher needed to create a lot of services to perform whatever punctually properly!

  16. Fiona says:

    Hey. I usually cannot want reading site writing because I feel that they are poor quality. Nevertheless, that 1 was worth browsing because it provides worthwhile and inspected information.

  17. Kaleigh says:

    Hi here everyone! I am Kaleigh and I prefer your web-page and specifically this article. I need to see something more. Can I compose to you a individual email, plz?

  18. Karina says:

    Hi. I'm Karina. I am just your average audience. I am satisfied of your methods – indeed, i am proud of you that you write so awesome writing and I could see them. Done well!

  19. Effie says:

    Hello everybody! It's very enjoyable review. I'm astonished that you are so gifted. I didn't esteem you!

  20. Melanie says:

    Congratulation on publishing incredible information! I need think that the article is a maximum quality text and it meets all kinds of standards. Well-done.

  21. Andrea says:

    Hey. Here is a question for everyone who try to find big excellence reports. Will you seek different website? If you consult me I will state NO. That page is perfect for myself!

  22. Holly says:

    It really is very challenging to compose a professional texts in a brief time period and still posses brains full of concepts to make new ones. I must state that the bloger is a pro writer.

  23. Ruby says:

    Right now here is a brief problem to ask – how to produce this sort of great article? It has almost all factors what make a close content. Here is a getting label and interesting articles. Well done!

  24. Fiona says:

    I have got to acknowledge that truly one of the most interesting articles I have ever understand. It's also skilfully developed. I am an English instructor so I know what I mean.

  25. DB Project Expert says:

    Only Professional Writers Can Make This Kind Of Material, Cheers

  26. Sandy James@ExerciseN says:

    Growing fibers is something that you can't control. It is highly likely that you keep moving forward and make the most of the things you have in life.

  27. Assignment Help says:

    You should have clearly define topics which you want to learn and start collecting study material according to that topics. Also some assignment help service you can use to get knowledge as well. Just proceed accordingly and don’t waste your time because time is precious.

  28. Doctor Who Costumes says:

    POSIX Portable Operating System Interface is an arrangement of standard working framework interfaces dependent on the Unix working framework. These are the primary two interfaces, however extra interfaces, for example, POSIX.4 for string administration, have been produced or are being created.

  29. Essay Help says:

    If you want to get any sort of writing assistance for any subject, the PhD writers at EssayAssignmentHelp is here to help you out.

  30. Puzzle Games says:

    Fun solution of the Puzzle Games.. If you are looking for a challenge, it’s time to check them out! https://games.lol/puzzle/

  31. Torrent Proxy says:

    The growing fibers are the best thing which has been seen in this thing for the people.

  32. college essay help online says:

    You have shared a very informative post which I was searching for a long time. Thanks for helping us to improve our knowledge by your precious information.

  33. buy essay online says:

    Really this article is very useful and informative for everyone. And I hope you'll share more ideas and useful article with us.

  34. Tuition assignments Singapore says:

    I have heard a lot about your contents and the great stories it shared. Please keep on posting.

  35. Assignment Help says:

    The above information you shared give me a new look at this topic. I am working as a full-time academic writer in myassignmentHelpsg providing Assignment Help Singapore services to college students.

  36. imgrum says:

    Thank you for sharing the post~ This is what I need to find.

  37. stephanie Micheal says:

    you can buy best leather jackets from here.

  38. Punjabi Song Lyrics says:

    New English Songs Download- Listen 2019 New English songs free online or Download Latest English Songs MP3.

  39. Assignment Experts says:

    Assignment experts use impressive language to make the assignment rich in quality. They proofread and edit the write-up to guarantee that there are no silly mistakes in the assignment.

  40. Online GCSE Coursework Writing Service says:

    This service is of high quality and provides you with some of the best results for the entire student population. In fact, it negates the morale of cheating when little help is available to students.

  41. website designing company in patna says:

    I am grateful to the owner of this site which really shares this wonderful work of this site.That is actually great and useful information.I'm satisfied with just sharing this useful information with us. Please keep it up to date like this.Thank you for sharing..

  42. packers and movers in patna says:

    I think this is one of the best blog for me because this is really helpful for me. Thanks for sharing this valuable information for free...

  43. cctv camera dealers in patna says:

    Nice information. Thanks for sharing this informative blog with us. I really need this type of blog and I’m so lucky to found this...

  44. ddjjatt says:

    I have been a keen follower of your website.
    recently I came across this topic and after reading the whole article I am amazed that how well you have written it.

  45. Essay Assignment Help says:

    Are you facing difficulty in handling your essay assignment? Are the deadlines troubling you? If you need help with essay assignment, get in touch with GotoAssignmentHelp. The assignments prepared by our dedicated writers are of premium and high quality.

  46. Mitchell Robinson says:

    Very informative post.

  47. Website Designing Company in Delhi says:

    Your method is really amazing to manage the words in your and such valuable information with us. Get the designing and development services by the best Website Designing Company in Delhi at Ogen Infosystem.

  48. Lifestyle Magazine says:

    Nice blog, keep more update about this related topic. Visit our website for lifestyle magazine.

  49. facebook marketing services says:

    Visit our website for lifestyle magazine.

  50. digital marketing company says:

    this is something great

  51. garage door repair alexandria va says:

    Nice blog, keep more update about this related topic.

  52. guilin tours says:

    Visit Guilin Once in a year

  53. The Soft Logix says:

    Really great information. I always love to read and spread this kind of information that is unique and really informative. Keep up the good work.

  54. Led Sign Lights says:

    Thanks for sharing great information. I always love to read and spread this kind of information that is unique and really informative.

  55. Mutual Fund Companies says:

    Nice blog, Get the Mutual Fund Schemes by Investment firms and Mutual Fund Companies.

  56. Bulk SMS says:

    Seamless communication is what businesses aim for to have an active and healthy relationship with customers. For this, they need a swift and effective channel and nothing can beat bulk SMS in this regard. Text messaging is not going anywhere as it is still the fastest and most preferred channel.

  57. Putlocker says:

    Looking forward to examining more. Good post. Actually thanks!

  58. SolarMovie says:

    Appreciated every bit regarding your blog post post. Cheers Again.

  59. Fmovies says:

    I enjoy you sharing that blog article.Really thanks!

  60. assignment help online says:

    If the students who are seeking any sort of professional help for their assignment can contact the Assignment Studio website. The assignment work that is provided by the Assignment Studio writers meet high quality standards and is free from plagiarism.

  61. Do my law assignment says:

    Hey, Your blog is very informative. It is nice to read such high-quality content. Attractive information on your blog, thank you for taking the time and share with us. Myassignmenthelp delivers high quality content related to do my law assignment.

  62. explosederire.com says:

    Hindi

  63. power washing says:

    Wow, I love your post today! This in just the perfect time. Thanks a bunch!

  64. heart in New Jersey says:

    Heart Sticker designs from the creator of the Heart in USA, Heart in (State), Bigfoot, Rebus Puzzle and many other designs. Many of our designs are also available on T-shirts. I invite you to check out our site.

  65. Lifestyle Magazine India says:

    Decent information, thank you so much for sharing with us. Visit Lifestyle magazine for parties and events in Delhi, India.

  66. Podayos says:
  67. le chateau guillestre says:
  68. windows app development company says:

    Windows app development is evolving a lot in the present years and it becomes an essential element for the progress and development of any business. Now you can get help from windows app development company and they will work in today’s modern technical world. And they will provide you 360 degrees development solutions.

  69. Coursework Help says:

    The assignment help given by our devoted authors are of high caliber. Are the deadlines troubling you? If you need homework help, get in touch with GotoAssignmentHelp. Our proficient and expert writers at GotoAssignmentHelp provide an unmatchable and exceptional coursework help service to help students realize their academic potential and fetch top grades.

  70. aditya sharma says:

    Wondermouse Technologies is the best Web Development Company in gurgaon which has spent significant time in advertising and building websites for wide range of business experts. At WonderMouse Technologies, we pride ourselves on building quality websites that successfully showcase your aptitude to create new customers, which makes us the best in designing the website for our clients hence, WonderMouse is also one of the top graphic design company , with a wide range of clients, keeping their satisfaction on the top of our priority list.

  71. Homework Help says:

    The assignment help given by our devoted authors are of premium and high caliber. If you need assignment help Australia, get in touch with GotoAssignmentHelp. We can guarantee you that your scholastic potential will apparently be enhanced and you'll get fantastic grades in your thesis help.

  72. montres 24-heures says:
  73. ESSAY WRITING SERVICE UK says:

    Deadlines for assignments are giving you sleepless nights? Get the best essay help discounts and offers by using online essay helper at moderate costs just with GotoEssayHelp Company. We have in-house pros and experts with great experience and knowledge in various fields which can give you best essay writing service UK.

  74. impressbss says:

    Like!! Really appreciate you sharing this blog post.Really thank you! Keep writing.

  75. yas says:
  76. Web Design Company says:

    Keep it up for more updates about this related blog. For a responsive and creative website designing and also for digital marketing services, visit Ogen Infosystem Delhi, India.

  77. fredluis says:

    Fantastic precise info and this is one of the most nice blogs Ive read in a very long time. carpet cleaner

  78. dissertation statistics help says:

    Good post 1

  79. Agence web Tunisie says:

    Hi, Excellent tutorial article. I love to read it

  80. agence de location de voiture en Tunisie says:

    Great Article… Simple and best guide…

  81. data analysis services says:

    Thanks for posting this info. Kindly share more such articles so that I can get better insight.statistical consulting Firms

  82. Best IT companies in India says:

    rel='external nofollow'nice blog

  83. School Management System says:

    Thanks for sharing this

  84. Martin luthar says:

    You are one step away from saving your Apple device from various issues or problems. Simply make contact with Apple Support team associates and resolve your entire issues related to Apple device. At here, I and my whole team is available 24x7 to help you in eliminating apple device issues from the root.

  85. Homework Help Canada says:

    Assignment help is an organization where students can get increasingly inconceivable plans in which we can get immediate contact with the specialists who can compose the task as indicated by the client with full fixation.

  86. heart in Oregon says:

    Famous for the green Heart in Oregon sticker. Also, Heart in America, Heart in your home state, Bigfoot, Yeti & Heart Sticker designs.

  87. Marketing Assignment Help Online says:

    Thanx For Sharing Such Useful Post Keep It Up :)

  88. School Management System says:

    Thanks for sharing this

  89. temple run 2 says:

    I just joined the forum so there are so many things I don't know yet, I hope to have the help of the boards, and I really want to get to know you all on the forum.

  90. online payday loans in Alabama says:

    Great write-up, I am a big believer in commenting on blogs to inform the blog writers know that they’ve added something worthwhile to the world wide web!

  91. Assignment help says:

    Thanks for sharing this marvelous post. I m very pleased to read this article.

  92. canlı tv izle says:

    Canlı TVS olarak sizlere HD kalitesindeki tüm televizyon yayınlarını kesintisiz bir şekilde sunuyoruz.Canlı tv izle butonlarına tıkladığınız andan itibaren beğendiğiniz ve takip ettiğiniz kanalın yayın akışına anlık olarak ulaşabilirsiniz.

  93. CERTIFICATE TRANSLATION SERVICES SINGAPORE says:

    I really enjoy your article. Thank you so much for sharing these amazing tips I must say you are an incredible writer, I love the way you describe things. Please keep sharing.

  94. coupon outlet says:

    I just visit this website by mistake but when I see the article I am very satisfied this it is goodcoupon outlet

  95. Crackslive.com says:

    Nicely this article that i've been waited for so long. it has exact same topic together with your write-up. Thanks, good share

  96. appliance repair Orlando says:

    I’m glad that you shared this helpful info with us.

  97. online assignment help says:

    ABC Assignment Help provides professional assignment help in various subjects for students studying in colleges and Universities across Australia, UK, USA, New Zealand and Canada. We deliver quality solutions for assignments in over 100 subjects. Our tutors focus on providing step-by-step solution to every assignment problem.

  98. My assignment help says:

    My Assignment Help is the most preferred professional assignment help service providing support to students across the globe. With our team of knowledgeable and well-trained writers we offer best assignment solutions coming with a guarantee of 100% original and impressive write-ups.

  99. Online Payday Loans Denver says:

    Search for payday loans near me and get instant approval payday loans Denver. Apply for online payday loans Colorado Springs no credit check. Visit coloradoloansnearme.com, a valuable service provider for your all financial needs.

  100. candy crush soda says:

    I have a lot of knowledge from what you share, to say thank you, the information and knowledge here helps me a lot

  101. Australia Assignment Help says:

    High study students they tensed about assignment writing work. Worry no more, we provide MBA assignment help services and any subjects assignment help services. We have 15+ years of experienced experts they help you anytime with cheap price. Hire Now!

  102. High Bun says:

    Webfeed360 is online media website which explore the trending news & true stories around the world, Here you can fulfill your daily dose on Entertainment.

  103. andrew symond says:

    To complete your thesis report, hire the Singapore Ph.D. Expert thesis writers and know today about our homework help services from our Ph.D. Experts. A team of professional homework writers who are committed to delivering the finest Ph.D. homework writing help for the Singapore University students.

  104. photographia says:

    Great post!! Thanks to share this best information with.

Leave a Reply