wingolog

lightweight concurrency in lua

16 May 2018 3:17 PM (concurrency | lua | fibers | guile | concurrent ml | coroutines | delimited continuations | igalia | snabb)

Hello, all! Today I'd like to share some work I have done recently as part of the Snabb user-space networking toolkit. Snabb is mainly about high-performance packet processing, but it also needs to communicate with management-oriented parts of network infrastructure. These communication needs are performed by a dedicated manager process, but that process has many things to do, and can't afford to make blocking operations.

Snabb is written in Lua, which doesn't have built-in facilities for concurrency. What we'd like is to have fibers. Fortunately, Lua's coroutines are powerful enough to implement fibers. Let's do that!

fibers in lua

First we need a scheduling facility. Here's the smallest possible scheduler: simply a queue of tasks and a function to run those tasks.

local task_queue = {}

function schedule_task(thunk)
   table.insert(task_queue, thunk)
end

function run_tasks()
   local queue = task_queue
   task_queue = {}
   for _,thunk in ipairs(queue) do thunk() end
end

For our purposes, a task is just a function that will be called with no arguments.

Now let's build fibers. This is easier than you might think!

local current_fiber = false

function spawn_fiber(fn)
   local fiber = coroutine.create(fn)
   schedule_task(function () resume_fiber(fiber) end)
end

function resume_fiber(fiber, ...)
   current_fiber = fiber
   local ok, err = coroutine.resume(fiber, ...)
   current_fiber = nil
   if not ok then
      print('Error while running fiber: '..tostring(err))
   end
end

function suspend_current_fiber(block, ...)
   -- The block function should arrange to reschedule
   -- the fiber when it becomes runnable.
   block(current_fiber, ...)
   return coroutine.yield()
end

Here, a fiber is simply a coroutine underneath. Suspending a fiber suspends the coroutine. Resuming a fiber runs the coroutine. If you're unfamiliar with coroutines, or coroutines in Lua, maybe have a look at the lua-users wiki page on the topic.

The difference between a fibers facility and just coroutines is that with fibers, you have a scheduler as well. Very much like Scheme's call-with-prompt, coroutines are one of those powerful language building blocks that should rarely be used directly; concurrent programming needs more structure than what Lua offers.

If you're following along, it's probably worth it here to think how you would implement yield based on these functions. A yield implementation should yield control to the scheduler, and resume the fiber on the next scheduler turn. The answer is here.

communication

Once you have fibers and a scheduler, you have concurrency, which means that if you're not careful, you have a mess. Here I think the Go language got the essence of the idea exactly right: Do not communicate by sharing memory; instead, share memory by communicating.

Even though Lua doesn't support multiple machine threads running concurrently, concurrency between fibers can still be fraught with bugs. Tony Hoare's Communicating Sequential Processes showed that we can avoid a class of these bugs by treating communication as a first-class concept.

Happily, the Concurrent ML project showed that it's possible to build these first-class communication facilities as a library, provided the language you are working in has threads of some kind, and fibers are enough. Last year I built a Concurrent ML library for Guile Scheme, and when in Snabb we had a similar need, I ported that code over to Lua. As it's a new take on the problem in a different language, I think I've been able to simplify things even more.

So let's take a crack at implementing Concurrent ML in Lua. In CML, the fundamental primitive for communication is the operation. An operation represents the potential for communication. For example, if you have a channel, it would have methods to return "get operations" and "put operations" on that channel. Actually receiving or sending a message on a channel occurs by performing those operations. One operation can be performed many times, or not at all.

Compared to a system like Go, for example, there are two main advantages of CML. The first is that CML allows non-deterministic choice between a number of potential operations in a generic way. For example, you can construct a operation that, when performed, will either get on one channel or wait for a condition variable to be signalled, whichever comes first. In Go, you can only select between operations on channels.

The other interesting part of CML is that operations are built from a uniform protocol, and so users can implement new kinds of operations. Compare again to Go where all you have are channels, and nothing else.

The CML operation protocol consists three related functions: try which attempts to directly complete an operation in a non-blocking way; block, which is called after a fiber has suspended, and which arranges to resume the fiber when the operation completes; and wrap, which is called on the result of a successfully performed operation.

In Lua, we can call this an implementation of an operation, and create it like this:

function new_op_impl(try, block, wrap)
   return { try=try, block=block, wrap=wrap }
end

Now let's go ahead and write the guts of CML: the operation implementation. We'll represent an operation as a Lua object with two methods. The perform method will attempt to perform the operation, and return the resulting value. If the operation can complete immediately, the call to perform will return directly. Otherwise, perform will suspend the current fiber and arrange to continue only when the operation completes.

The wrap method "decorates" an operation, returning a new operation that, if and when it completes, will "wrap" the result of the completed operation with a function, by applying the function to the result. It's useful to distinguish the sub-operations of a non-deterministic choice from each other.

Here our new_op function will take an array of operation implementations and return an operation that, when performed, will synchronize on the first available operation. As you can see, it already has the equivalent of Go's select built in.

function new_op(impls)
   local op = { impls=impls }
   
   function op.perform()
      for _,impl in ipairs(impls) do
         local success, val = impl.try()
         if success then return impl.wrap(val) end
      end
      local function block(fiber)
         local suspension = new_suspension(fiber)
         for _,impl in ipairs(impls) do
            impl.block(suspension, impl.wrap)
         end
      end
      local wrap, val = suspend_current_fiber(block)
      return wrap(val)
   end

   function op.wrap(f)
      local wrapped = {}
      for _, impl in ipairs(impls) do
         local function wrap(val)
            return f(impl.wrap(val))
         end
         local impl = new_op_impl(impl.try, impl.block, wrap)
         table.insert(wrapped, impl)
      end
      return new_op(wrapped)
   end

   return op
end

There's only one thing missing there, which is new_suspension. When you go to suspend a fiber because none of the operations that it's trying to do can complete directly (i.e. all of the try functions of its impls returned false), at that point the corresponding block functions will publish the fact that the fiber is waiting. However the fiber only waits until the first operation is ready; subsequent operations becoming ready should be ignored. The suspension is the object that manages this state.

function new_suspension(fiber)
   local waiting = true
   local suspension = {}
   function suspension.waiting() return waiting end
   function suspension.complete(wrap, val)
      assert(waiting)
      waiting = false
      local function resume()
         resume_fiber(fiber, wrap, val)
      end
      schedule_task(resume)
   end
   return suspension
end

As you can see, the suspension's complete method is also the bit that actually arranges to resume a suspended fiber.

Finally, just to round out the implementation, here's a function implementing non-deterministic choice from among a number of sub-operations:

function choice(...)
   local impls = {}
   for _, op in ipairs({...}) do
      for _, impl in ipairs(op.impls) do
         table.insert(impls, impl)
      end
   end
   return new_op(impls)
end

on cml

OK, I'm sure this seems a bit abstract at this point. Let's implement something concrete in terms of these primitives: channels.

Channels expose two similar but different kinds of operations: put operations, which try to send a value, and get operations, which try to receive a value. If there's a sender already waiting to send when we go to perform a get_op, the operation continues directly, and we resume the sender; otherwise the receiver publishes its suspension to a queue. The put_op case is similar.

Finally we add some synchronous put and get convenience methods, in terms of their corresponding CML operations.

function new_channel()
   local ch = {}
   -- Queues of suspended fibers waiting to get or put values
   -- via this channel.
   local getq, putq = {}, {}

   local function default_wrap(val) return val end
   local function is_empty(q) return #q == 0 end
   local function peek_front(q) return q[1] end
   local function pop_front(q) return table.remove(q, 1) end
   local function push_back(q, x) q[#q+1] = x end

   -- Since a suspension could complete in multiple ways
   -- because of non-deterministic choice, it could be that
   -- suspensions on a channel's putq or getq are already
   -- completed.  This helper removes already-completed
   -- suspensions.
   local function remove_stale_entries(q)
      local i = 1
      while i <= #q do
         if q[i].suspension.waiting() then
            i = i + 1
         else
            table.remove(q, i)
         end
      end
   end

   -- Make an operation that if and when it completes will
   -- rendezvous with a receiver fiber to send VAL over the
   -- channel.  Result of performing operation is nil.
   function ch.put_op(val)
      local function try()
         remove_stale_entries(getq)
         if is_empty(getq) then
            return false, nil
         else
            local remote = pop_front(getq)
            remote.suspension.complete(remote.wrap, val)
            return true, nil
         end
      end
      local function block(suspension, wrap)
         remove_stale_entries(putq)
         push_back(putq, {suspension=suspension, wrap=wrap, val=val})
      end
      return new_op({new_op_impl(try, block, default_wrap)})
   end

   -- Make an operation that if and when it completes will
   -- rendezvous with a sender fiber to receive one value from
   -- the channel.  Result is the value received.
   function ch.get_op()
      local function try()
         remove_stale_entries(putq)
         if is_empty(putq) then
            return false, nil
         else
            local remote = pop_front(putq)
            remote.suspension.complete(remote.wrap)
            return true, remote.val
         end
      end
      local function block(suspension, wrap)
         remove_stale_entries(getq)
         push_back(getq, {suspension=suspension, wrap=wrap})
      end
      return new_op({new_op_impl(try, block, default_wrap)})
   end

   function ch.put(val) return ch.put_op(val).perform() end
   function ch.get()    return ch.get_op().perform()    end

   return ch
end

a wee example

You might be wondering what it's like to program with channels in Lua, so here's a little example that shows a prime sieve based on channels. It's not a great example of concurrency in that it's not an inherently concurrent problem, but it's cute to show computations in terms of infinite streams.

function prime_sieve(count)
   local function sieve(p, rx)
      local tx = new_channel()
      spawn_fiber(function ()
         while true do
            local n = rx.get()
            if n % p ~= 0 then tx.put(n) end
         end
      end)
      return tx
   end

   local function integers_from(n)
      local tx = new_channel()
      spawn_fiber(function ()
         while true do
            tx.put(n)
            n = n + 1
         end
      end)
      return tx
   end

   local function primes()
      local tx = new_channel()
      spawn_fiber(function ()
         local rx = integers_from(2)
         while true do
            local p = rx.get()
            tx.put(p)
            rx = sieve(p, rx)
         end
      end)
      return tx
   end

   local done = false
   spawn_fiber(function()
      local rx = primes()
      for i=1,count do print(rx.get()) end
      done = true
   end)

   while not done do run_tasks() end
end

Here you also see an example of running the scheduler in the last line.

where next?

Let's put this into perspective: in a couple hundred lines of code, we've gone from minimal Lua to a language with lightweight multitasking, extensible CML-based operations, and CSP-style channels; truly a delight.

There are a number of possible ways to extend this code. One of them is to implement true multithreading, if the language you are working in supports that. In that case there are some small protocol modifications to take into account; see the notes on the Guile CML implementation and especially the Manticore Parallel CML project.

The implementation above is pleasantly small, but it could be faster with the choice of more specialized data structures. I think interested readers probably see a number of opportunities there.

In a library, you might want to avoid the global task_queue and implement nested or multiple independent schedulers, and of course in a parallel situation you'll want core-local schedulers as well.

The implementation above has no notion of time. What we did in the Snabb implementation of fibers was to implement a timer wheel, inspired by Juho Snellman's Ratas, and then add that timer wheel as a task source to Snabb's scheduler. In Snabb, every time the equivalent of run_tasks() is called, a scheduler asks its sources to schedule additional tasks. The timer wheel implementation schedules expired timers. It's straightforward to build CML timeout operations in terms of timers.

Additionally, your system probably has other external sources of communication, such as sockets. The trick to integrating sockets into fibers is to suspend the current fiber whenever an operation on a file descriptor would block, and arrange to resume it when the operation can proceed. Here's the implementation in Snabb.

The only difficult bit with getting nice nonblocking socket support is that you need to be able to suspend the calling thread when you see the EWOULDBLOCK condition, and for coroutines that is often only possible if you implemented the buffered I/O yourself. In Snabb that's what we did: we implemented a compatible replacement for Lua's built-in streams, in Lua. That lets us handle EWOULDBLOCK conditions in a flexible manner. Integrating epoll as a task source also lets us sleep when there are no runnable tasks.

Likewise in the Snabb context, we are also working on a TCP implementation. In that case you want to structure TCP endpoints as fibers, and arrange to suspend and resume them as appropriate, while also allowing timeouts. I think the scheduler and CML patterns are going to allow us to do that without much trouble. (Of course, the TCP implementation will give us lots of trouble!)

Additionally your system might want to communicate with fibers from other threads. It's entirely possible to implement CML on top of pthreads, and it's entirely possible as well to support communication between pthreads and fibers. If this is interesting to you, see Guile's implementation.

When I talked about fibers in an earlier article, I built them in terms of delimited continuations. Delimited continuations are fun and more expressive than coroutines, but it turns out that for fibers, all you need is the expressive power of coroutines -- multi-shot continuations aren't useful. Also I think the presentation might be more straightforward. So if all your language has is coroutines, that's still good enough.

There are many more kinds of standard CML operations; implementing those is also another next step. In particular, I have found semaphores and condition variables to be quite useful. Also, standard CML supports "guards", invoked when an operation is performed, and "nacks", invoked when an operation is definitively not performed because a choice selected some other operation. These can be layered on top; see the Parallel CML paper for notes on "primitive CML".

Also, the choice operator above is left-biased: it will prefer earlier impls over later ones. You might want to not always start with the first impl in the list.

The scheduler shown above is the simplest thing I could come up with. You may want to experiment with other scheduling algorithms, e.g. capability-based scheduling, or kill-safe abstractions. Do it!

Or, it could be you already have a scheduler, like some kind of main loop that's already there. Cool, you can use it directly -- all that fibers needs is some way to schedule functions to run.

godspeed

In summary, I think Concurrent ML should be better-known. Its simplicity and expressivity make it a valuable part of any concurrent system. Already in Snabb it helped us solve some longstanding gnarly issues by making the right solutions expressible.

As Adam Solove says, Concurrent ML is great, but it has a branding problem. Its ideas haven't penetrated the industrial concurrent programming world to the extent that they should. This article is another attempt to try to get the word out. Thanks to Adam for the observation that CML is really a protocol; I'm sure the concepts could be made even more clear, but at least this is a step forward.

All the code in this article is up on a gitlab snippet along with instructions for running the example program from the command line. Give it a go, and happy hacking with CML!

165 responses

  1. Solitaire Card Game Free says:

    I am big fan for this card game and am always satisfied to play solitaire times, everyone to play in without registration and download.

  2. Business Logo Design says:

    Tarantool is a strong open-source elective that has demonstrated itself at scale. Its best highlights are a full Lua application server that keeps running by its database server, its capacity to work with informational indexes bigger than RAM (empowered by its Vinyl motor) and its fiber demonstrate for simultaneous.

  3. proprepandfulfillment says:

    I want to know about it from a to z, I mean from beginning to end. I need every single information about this and want to write a book on this topic...

  4. https://www.reddit.com/r/Pollowers/ says:

    The desire to make money online through the use of social platforms is perfectly normal. In order to achieve this quest, an optimal blogging system is required. A sound blogging system would allow its user adopt a systematic approach to quench his blogging endeavors.

  5. Write an Essay For Me says:

    Parallelism essentially implies that everything in the sentence must match. A more formal definition might be parallelism is the utilization of progressive verbal developments in verse or exposition that compare in syntactic structure, sound, meter, which means, and so forth.

  6. Arne Babenhauserheide says:

    Do you have a performance benchmark of the lua-fibers?

    How does it look when you pit it against the other fiber implementations in the skynet-benchmark? → https://github.com/atemerev/skynet

  7. Do Essay says:

    I really like your work and also I like this website because both things are awesome and amazing. You worked really hard and all readers should learn from your thoughts.

  8. assignment help service says:

    There are widespread anarchist philosophies, such as anarchocommunism, which reject and try to find alternatives to both currency and exchange, but it's not as if these philosophies forbid currency or exchange.

  9. aqoo maama says:

    , but it's not as if these philosophies forbid currency or exchange.

  10. British Assignment Writer says:

    The light-weight concurrency era is the following step to evolve programming languages to the multi-core age. this will allow packages to take gain of more than eight cores the use of task-primarily based algorithms and lowering the concurrency overhead.

  11. Game hacker says:

    game hacker download

  12. Pandora One APK says:

    download pandora one apk

  13. Game Hacker says:

    Download SB Game Hacker APK

  14. Mobilism APK says:

    download mobilism apk

  15. Blackmart Alpha says:

    download blackmart alpha

  16. pcsx2 download says:

    pcsx2 download

  17. gangstar vegas says:

    download gangstar vegas

  18. Happy Diwali 2018 says:

    I found a lot of interesting information here.

  19. how to make slime says:

    Ahh that is great thank you ! Good for special needs too !

  20. Diwali Images 2018 says:

    that is great thank you !

  21. pandora apk says:

    I have found good information here

  22. Assignment help online uk says:

    I read the whole article but didn't understand anything in this and I also searched on google about lightweight concurrency in Lua can anyone tell me about this whole article I wanted to about this?

  23. Manuel Gonzalez says:

    We have perfect vacation rentals and property management LLC in Cape Coral. If Your are looking for Real Estate Broker and a Certified Property Manager contact Us 239-204-7384

  24. The Oscars 2019 Live stream says:

    If Gold Derby’s combined Oscar odds for the Best Picture category reflect reality, six out of the top 10 contenders fit the mold of a biopic. They include “Roma,” “Green Book,” “The Favourite,” “BlackKklansman,” “Vice” and “First Man.”

  25. kahootspam says:

    very strong captcha

  26. Digital Health Trend says:

    I want to know about it from a to z, I mean from beginning to end. I need every single information about this and want to write a book on this topic...

  27. Walton Hall says:

    No matter how intelligent you are, there would be a time when you might have to visit a writing service. Visit us and have some time for yourself.

  28. George says:

    I had to do my essays online recently and your website really helped me out with searching for information. Thank you for posting and sharing this :)

  29. www.assignmenthelpfolks.com says:

    Thanks for your great effort as you described here smallest possible scheduler for simply a queue of tasks and a function to run those tasks.

  30. Online Dissertation Writing Service says:

    your concept of lightweight concurrency in lua is great I am very happy to read your article and ver I learn many things in your blog.

  31. we says:

    Wow, that's impressive!)

  32. Port Antigua Rentals says:

    Amazing work by author

  33. John says:

    https://www.heathcafe.net/home-remedies-for-skin-boils/
    https://www.heathcafe.net/reduce-in-3-days-heavyweight/
    https://www.heathcafe.net/fat-burning-workout-programs/
    https://www.heathcafe.net/tired-eyes-from-work/
    https://www.heathcafe.net/
    https://www.heathcafe.net/eat-cashew-nuts-or-almonds/
    https://www.heathcafe.net/spinal-cord-problem/
    https://www.heathcafe.net/hair-care-tips/
    https://www.heathcafe.net/easy-tips-for-fast-weight-loss/
    https://www.heathcafe.net/can-neem-cancer-stop-it/

  34. Walmond Zack says:

    The choice to make cash online via the use of social structures is perfectly ordinary. So one can attain this quest, an ideal blogging device is required. The light-weight concurrency generation is the subsequent step to evolve programming languages to the multi-middle age. Professional Assignment Writing Service

  35. Ajay says:

    Digital marketer in ahmedabad

  36. https://www.lampungservice.com says:

    ..

  37. https://bateraitanam.blogspot.com says:

    ..

  38. https://tempatservicehpdibandarlampung.blogspot.com/ says:

    ...

  39. uprise kumar says:

    Everyone know the pandora one apk is best application for entertainment And this app really a good application

  40. IALM says:

    Ialm is the best online law courses platform who want to learn online

  41. bapasitaramprints says:

    Saree is one of the oldest traditional clothing in India. Low-cost Indian attire is one that fits the budget of everyone. The designer sarees is one of the most popular ethnic fashion wears for Indian women today. Over the years, it has only become more appealing, stylish and all-time favorites for women sarees Wholesalers in Surat.

  42. customessaysservice says:

    It is a good article. It is good to know regarding the changes in the currency. These changes are very helpful in the development of the country. It is really easy to carry these currencies for the citizens. These changes all are to be done for the welfare of the citizens. It is economical so it will also help the government to save money in the making of this type of currency. It is a great example for all countries.

  43. bluesky says:

    nice article and very useful

  44. bluesky says:

    This site is very attractive and informatics

  45. dissertation writing services says:

    I have seen many useful elements on your web site about personal computers. However, I’ve the thoughts and opinions that notebook

  46. Assignment Help Services says:

    The preference to make money online through the usage of social platforms is flawlessly ordinary. in an effort to acquire this quest, a premier blogging machine is required. You worked truly tough and all readers should examine out of your thoughts. https://excellentacademichelp.com/assignment-writing/

  47. Writer-elite says:

    Stuck with annoying or boring tasks? There are many options. First is go to the library and make your assignments. And another option is to rely on professional writers. Take a look at Writer-elite service to find out more.

  48. sprinkler contactor says:

    I’ve been visiting your blog for a while now and I always find a gem in your new posts. Thanks for your usual wonderful effort.

  49. bluesky says:

    mind-blowing information sahring

  50. mobilism download says:

    I’ve been visiting your blog for a while now and I always find a gem in your new posts. Thanks for your usual wonderful effort.

  51. Montre festina says:

    http://montre-24h.com/festina/

  52. up drive says:

    nice post

  53. cheap coursework help online says:

    Hello to everyone! Nowadays writing papers is one of the main tasks for students. But with the help of the writing service, you can make this process easier and quicker. Also, a big advantage of using such service is that papers written rightly!

  54. Alison Macgomery says:

    Whatever field you want to be a genius in already has a lot of exceptional people in it. Find them, talk to them, become their best friends if you can here https://essaysprofessors.com/short-answer-questions.html . The more you are exposed to other people’s ideas and way of going about things, the more you will adopt ideas and ways of going about things to help you excel in your area.

  55. Olivia Green says:

    Reaction paper means explaining your attitude and thoughts towards some novel, book item. To do this properly, you need to allow sufficient time to read and digest the content in order to combine your thoughts and ideas. https://primeessays.com/reaction-paper.html

  56. Florida Vacation Rentals says:
  57. click here says:

    I’ve been visiting your blog for a while now and I always find a gem in your new posts. Thanks for your usual wonderful effort.

  58. remodeling says:

    You have brought up a very superb points , regards for the post.

  59. David Dutton says:

    nice thanks for share this.

  60. happy wheels says:

    This is a great thing, I think everyone feels this information is very valuable, thank you

  61. Pandora Hacked Apk says:

    Nice! This information was inspiring for the readers of this blog

  62. Spotify Cracked Apk says:

    Spotify is one of the best platforms for listening to your favorite songs. Download the app on your phone right now

  63. adamschule85 says:

    I actually found several ones that I think will be useful for me. little alchemy hints

  64. Homeaway Ranch says:

    This is nice spot for for getting information Wild Life ranch borne texas hill country

  65. sia chals says:

    Nice blog I really appreciate your words, Dehradun escorts want to talk dirty and you are ready to go deep down. But, wait! They will never allow you to go to the main action so early. They know how to raise the curiosity level of males and want to make you feel that they are not easy to handle. if you want entrainment service full night visit

  66. sia chals says:

    Nice blog I really appreciate your words, Dehradun escorts want to talk dirty and you are ready to go deep down. But, wait! They will never allow you to go to the main action so early. They know how to raise the curiosity level of males and want to make you feel that they are not easy to handle. if you want entrainment service full night visit

  67. Hold Sneaker says:

    amplitude de mouvement de 215 $ à 330 $. Tout comme le créateur George Third. Troisième. Martin 's 6ème et 7ème guides, il ya certainement des chaussures chaussures pas chere ou des bottes tout simplement pas un signe de votre temps de lancement de chaussures de sport ou de bottes sur adidas. Com mais.

    Nous avons tous affaire à nos propres chaussures de jogging avec beaucoup de choses: un long chemin en ce qui concerne le tarmac graveleux; surfaces hors route cahoteuses; sols sales chaussures ou bottes sentiers de randonnée; et aussi des flaques pauvres en ce qui concerne les boissons métropolitaines hors route ou peut-être non définies. Oof. De plus, nous payons tous beaucoup pour des chaussures à faible coût ou des bottes spéciales.

  68. temple run 3 says:

    The information is very special, I will have to follow you, the information you bring is very real, reflecting correctly and objectively, it is very useful for society to grow together.

  69. shoeswebbutik says:

    dina distributörer vem som helst förbättrar ut och in i gymnastiksalen eftersom mls Under Armour Charged Reactor Run skor rea någon fack högt på din kurs. Dina In Foriegn-löparskor A är byggda för att tvinga vem som helst genom att ha en energiträning, men är emellertid en utmärkt separat mycket bra löpare i hela ett lämpligt. Uppstått på grund av att lättaste helt dämpade löpare i världen 229 g avseende storlek 8. 5, kan det vara mycket sneakers webbshop mer fast under basen jämfört med din nuvarande avera.

  70. Pandora Premium APK says:

    nice

  71. Pandora Premium APK says:

    apk apps for free

  72. spss help says:

    An understudy in the UK and need a best proficient help by then essentially visit at business help and take our proposition models ukat a sensible expense. Our top scholarly specialists will manage you towards completing your hazardous

  73. sewa mobil jakarta says:

    Nice article, thanks for the information.

  74. Ulric kaka says:

    As a business developer from the past 7 years, I have to build a lot of success full business throughout the world and currently working for Logosight as a business development officer and creating the best team.

  75. Assignment help says:

    Assignment help is the best resources for students.Allassignmenthelp provides best service in whole world. If you have any query related to this topic you can take assistance from assignment help online.

  76. ADWEB STUDIO says:

    This is probably up there with my top ten most favorite blog posts I have ever read. WELL DONE! LOVED this.

  77. Digital Saad says:

    Thanks for your great post. I like this very much, please write more about these, wait for your update.

  78. Assignment Helps says:

    The data is extremely uncommon, I should tail you, the data you bring is genuine, reflecting accurately and unbiasedly, it is helpful for society to become together. assignment helps

  79. UK Assignment Writers says:

    Pick students assignment help, assignment help UK services at an affordable price to get A+ grades from the professional UK assignment writers.

  80. Thames D says:

    This a new frontier in programming, quite some good work done here. For such and other programming homework help, find us here.

  81. Wizard says:

    Really a good post you shared with us . Thanks

  82. Ame lia says:

    Oh, send them your heart
    So they know that someone cares
    And their lives will be stronger and free
    As God has shown us by turning stones to bread
    And so we all must lend a helping handemoji

  83. fallout shelter cheats says:

    amazing blog i love it

  84. dissertation helps says:

    Astonishing article. Intriguing to examine. I really love to examine such a conventional article. Thankful! keep shaking ..dissertation helps

  85. Gta 5 cheats says:

    if you want to see the list of latest gta 5 cheats then visit the link

  86. Carl johnson says:

    If you are looking gta 4 cheats for any device then visit this awesome website - gta 4 cheats

  87. dissertation helps says:

    That is surprising.. I saw that minute when Ghost Rider came to consider astound significant. dissertation helps

  88. Cyber Abhi says:

    want some awesome apk ? follow this siteCyber Abhi

  89. Anna Stephen says:

    Thai is fundamental for me and solid for other people. Post is unfathomably clear and center the essential concerns. I need this sort ordinarily for learning. Extraordinary articles and perceive is amazingly novel. https://researchproposals.co.uk/

  90. hobbyladies says:

    Interesting place for chatting with the ladies on hobby ladies

  91. gtacheatcode says:

    Grab these amazing and Latest 2020 eu4 console commands

  92. Spotify APK says:

    Very Informative post.

  93. Diabetic Supplies Ann Arbor says:

    This discussion is very useful and informative.

  94. alen says:

    Love trying new music, check best tracks on pandora premium apk

  95. Contractor Oshawa says:

    Cool information here but not sure if I can apply it to my website

  96. Beyhadh 2 says:

    Hi, constantly i used to check weblog posts here in the early hours in the morning, because i like to find out more and more. Yeh Rishta Kya Kehlata Hai

  97. get assignment help online says:

    At getassignmenthelponline.comwe provide you the best and effective essay writing service. We help the seeking student according to their flexibility. Best Essay Writing Service

  98. Annie Jasmine says:

    You rock for amazing tips and data, couldn't require anything over to see more post from you right now. my crm system

  99. shell shockers says:

    The information you have posted is really good and very helpful, I will often visit your site.

  100. Trickyhunt says:

    Very Informative post. Thanks for sharing

  101. eu4 console commands says:

    Thanks for sharing this amazing information. I really enjoyed it. Keep up the good work and all the very best of luck! And if you have interest in games then visit gtacheatcode and click here for eu4 console commands .

  102. walmart near me says:

    I hope that there will be miracles, heal the broken heart is crying and sad. The sky is still divided night light right then we broke up already, so our love circle is like that.

  103. gtacheatcode says:

    Good day! I simply would like to give you a huge thumbs up for your great information you have here on this post. I am returning to your website for more soon. If you are looking for games cheats then visit – gtacheatcode and click here for gta 5 cheats ps4 .

  104. Research proposals says:

    https://anybloginfo.jimdofree.com/2018/07/04/the-camila-coelho-guide-to-summer-beauty-and-taking-selfies/

  105. Amelia Joseph says:

    Dazzling article with scrambling thought! Much thankfulness to you for such an essential article.It is really an objective and fundamental one, It truly help me to pass on my idea!! math assignments

  106. Cyber Abhi says:

    If you wish for an apk which gives you access to unlimited music then please follow this link Pandora One Mod Apk

  107. Yashavantha Yashu says:

    Very good content and good suggestions bro thank you so much

  108. Treasurebox online store nz says:

    Thanks for sharing this informative page.. Get you favourite items in Auckland Newzealand on Treasurebox store.

  109. Spotify Hacked Apk says:

    Cool post, thank you

  110. Jerry says:

    Thanks a lot for this info

  111. Chandoo says:

    If you want to improve your productivity you must have the best MacBook accessories that are most useful.

  112. Supreme mobiles says:

    Awesome postsuprememobiles

  113. e-ticaret danışmanlığı says:

    There’s definately a great info to know about about this issue. I like all the points you’ve made.

  114. sanal ofis says:

    I was just seeking this info for a while. After 6 hours of continuous Googleing, finally I got it in your website. I wonder what’s the lack of Google strategy that do not rank this kind of informative sites in top of the list. Generally the top sites are full of garbage.

  115. Finance Homework Help says:

    Great information. Very accurate and informative. I appreciate the author.

  116. Camper Abhi says:

    If you are into camping and hiking then you might be already knowing the importance of the best camping knives that is one of the important tools.

  117. gauri says:

    Are you looking for best mod apk paids app for free then you must visit our site for best 2020 free mods on Modlelo

  118. modlelo says:

    Are you a gamer and love to play games then Fortnite mods are just for you just to get all full mods for free just visit on https://modlelo.com/fortnite-mods/

  119. 3d animation studios nyc says:

    Thanks for sharing

  120. Online Essay Help says:

    You are sharing delightful post. It is effortless and easy to understand for me. Thanks for sharing the post.

  121. Assignment Help says:

    I really appreciate this type of working and interested to see more good posts in the coming future. Actually it's a very nice site! this is my first visit on this site my friend gave the link of this site! and the site is really great.

  122. Peaky Blinder Costumes says:

    I wanted to thank you for this websites! Thanks for sharing.

  123. Buy Essay Online Uk says:

    this article is genuinely a good paragraph, keep it up.

  124. Shopify developers says:

    we pride in offering high-end shopify developers and website development services for the extra mile.

  125. Time tracking app says:

    The simplest time tracker to help you get things done.

  126. it solution says:

    With 8 years of experience in the industry it solution has executed groundbreaking ideas for different purposes and spheres.

  127. Shopify app development says:

    Being a leading shopify app development platform, we provide optimal results by establishing a strong brand image across major search engines

  128. Dubai First SEO says:

    Nice Article. Thaks a lot for sharing this. SEO Expert

  129. Hydro Excavator Truck says:

    Thanks for sharing this article!

  130. army boots for rucking says:

    Awesome! Have you checked the best camping knives

  131. YouTube Vanced Apk says:

    Thank you so much for this wonderful info

  132. marlboro says:

    Thank you for such well-written information. It’s full of insightful information and entertaining descriptions. Your point of view is the best among many.

  133. marlboro says:

    Thank you for such well-written information. It’s full of insightful information and entertaining descriptions. Your point of view is the best among many.

  134. Men'S Leather Jackets says:

    When the temperature begins to drop, you draw out all the comfortable garments in an offer to remain comfortable in the nippy climate. You can get into all the sweaters and coats you need however then you would be settling on style.

  135. Digital Marketing Agency says:

    Thanks for sharing such great information, I highly appreciate your hard-working skills as the post you published have some great information which is quite beneficial for me, I hope you will post more like that in the future

  136. Digital Marketing Agency Dubai says:

    Digital marketing agencies in Dubai now exist in large numbers to provide several digitally based services, customer care, and other online facilities that will increase the profit gain at the end of the stock sale, however, it also might result in the highest production rate and to get a perceived advantage.

  137. Do My Economics Homework says:

    You must know what details help you write a good introduction. Firstly, you are required to discuss the contextual background of the topic.

  138. best blogger templates says:

    good work

  139. loud updates says:

    awesome work kee it up

  140. Ryan says:

    Nice post

  141. Mansi Negi says:

    Awsome article !!

  142. Assignment Help Uk says:

    You need to realize the significance of assignment help given by expert writers, and here we are revealing some key points which escort you to get advantage from New assignment help services.

  143. edgar says:

    This is valuable. Nice work. I highly recommend https://homeworkcheg.com/
    when looking for assignments.

  144. car engine parts uk says:

    The light-weight simultaneousness time is the accompanying advance to develop programming dialects to the multi-center age. this will permit bundles to take an increase of in excess of eight centers the utilization of undertaking essentially based calculations and bringing down the simultaneousness overhead.

  145. Automotive Engine says:

    Much obliged to you for such elegantly composed data. It's brimming with adroit data and engaging portrayals. Your perspective is the best among many.

  146. 4b Automotive says:

    Thankful to you for such carefully formed information. It's overflowing with skillful information and drawing in depictions. Your point of view is the best among many.

  147. geometry dash says:

    Thanks for sharing the information it was very helpful for me

  148. Bgrowth Ninja says:

    Thanks For Sharing Such A Great Post, Please Visit My Site To Learn What Is Direct Object Reflexive Pronouns

  149. mini militia mod apk says:
  150. mario kart tour apk says:
  151. engine rebuild kits says:

    Much obliged For Sharing Such A Great Post,

  152. Private Tutors says:

    I am so grateful for your blog.i Really looking forward to read more! Really Great.

  153. John says:

    HomeWorkForMe offers professional academic help for various academic levels, from high school to PhD. Click here for more and it will tell you more.

  154. face skin care says:

    A debt of gratitude is for your extraordinary exertion as you depicted here littlest conceivable scheduler for just a line of undertakings and a capacity to run those errands.

  155. Read online stories and poems says:

    ZenPens is an online publishing platform where you can read, write and publish short stories, poems, articles and novels. Just register as a writer and start publishing or if you are a reader, read on!

    https://bensomtoo.blogspot.com/p/links.html?m=1

  156. jake says:

    As a man, you should have the right to use free sex meetings for yourself with sex gratis. We offer you free sex today. So exactly what you were secretly looking for.

  157. privatehobbyhuren says:

    private hobbyhuren
    sollte sich jeder Mann mal gönnen. Denn es sind diese feuchten Hobbyhuren, die ihre flinken Finger und ihre Zungen nicht von den harten Schwänzen lassen können. Private Hobbyhuren gibt es hier wie Sand am Meer. Einfach mal reinschauen, die privaten Profile durchforsten und aus dem Staunen nicht mehr herauskommen. Denn hier wird einiges geboten.

  158. omasex says:

    Santa Barbara-born Zorica interests includes travelling, clothing fashion, kites. Also, she loves following a Ottawa senators icehockey-match live at the stadium.

  159. champ marketer says:

    eople and business interact with accounts which have more activity. Champ Marketer helps you get that traction!

  160. Steven says:

    thanks for sharing

  161. sextreff says:

    But where can guys best find these willing women for sex meetings?

  162. Tejar says:

    This is truly an practical and pleasant information for all. Thanks for sharing this to us

  163. Online Shopping Site in UAE says:

    Hey, Are you Looking for best online shopping site in UAE. Wigme is the best shopping site for you, here we provide all type of electronics products online.

  164. Pro dissertation Writing says:

    Our team of Best dissertation writing services in the UK is here to assist you throughout the process and get you done with your dissertation since we understand the complexity of it.

  165. Microwave Oven says:

    Looking for best Microwave Oven in UAE. Wigme is the best shopping site for you, here we provide all type of electronics products online.

Leave a Reply