wingolog

An incomplete history of language facilities for concurrency

12 October 2016 1:45 PM (pl | concurrency | erlang | go | csp | guile | fibers | callback hell)

I have lately been in the market for better concurrency facilities in Guile. I want to be able to write network servers and peers that can gracefully, elegantly, and efficiently handle many tens of thousands of clients and other connections, but without blowing the complexity budget. It's a hard nut to crack.

Part of the problem is implementation, but a large part is just figuring out what to do. I have often thought that modern musicians must be crushed under the weight of recorded music history, but it turns out in our humble field that's also the case; there are as many concurrency designs as languages, just about. In this regard, what follows is an incomplete, nuanced, somewhat opinionated history of concurrency facilities in programming languages, with an eye towards what I should "buy" for the Fibers library I have been tinkering on for Guile.

* * *

Modern machines have the raw capability to serve hundreds of thousands of simultaneous long-lived connections, but it’s often hard to manage this at the software level. Fibers tries to solve this problem in a nice way. Before discussing the approach taken in Fibers, it’s worth spending some time on history to see how we got here.

One of the most dominant patterns for concurrency these days is “callbacks”, notably in the Twisted library for Python and the Node.js run-time for JavaScript. The basic observation in the callback approach to concurrency is that the efficient way to handle tens of thousands of connections at once is with low-level operating system facilities like poll or epoll. You add all of the file descriptors that you are interested in to a “poll set” and then ask the operating system which ones are readable or writable, as appropriate. Once the operating system says “yes, file descriptor 7145 is readable”, you can do something with that socket; but what? With callbacks, the answer is “call a user-supplied closure”: a callback, representing the continuation of the computation on that socket.

Building a network service with a callback-oriented concurrency system means breaking the program into little chunks that can run without blocking. Whereever a program could block, instead of just continuing the program, you register a callback. Unfortunately this requirement permeates the program, from top to bottom: you always pay the mental cost of inverting your program’s control flow by turning it into callbacks, and you always incur run-time cost of closure creation, even when the particular I/O could proceed without blocking. It’s a somewhat galling requirement, given that this contortion is required of the programmer, but could be done by the compiler. We Schemers demand better abstractions than manual, obligatory continuation-passing-style conversion.

Callback-based systems also encourage unstructured concurrency, as in practice callbacks are not the only path for data and control flow in a system: usually there is mutable global state as well. Without strong patterns and conventions, callback-based systems often exhibit bugs caused by concurrent reads and writes to global state.

Some of the problems of callbacks can be mitigated by using “promises” or other library-level abstractions; if you’re a Haskell person, you can think of this as lifting all possibly-blocking operations into a monad. If you’re not a Haskeller, that’s cool, neither am I! But if your typey spidey senses are tingling, it’s for good reason: with promises, your whole program has to be transformed to return promises-for-values instead of values anywhere it would block.

An obvious solution to the control-flow problem of callbacks is to use threads. In the most generic sense, a thread is a language feature which denotes an independent computation. Threads are created by other threads, but fork off and run independently instead of returning to their caller. In a system with threads, there is implicitly a scheduler somewhere that multiplexes the threads so that when one suspends, another can run.

In practice, the concept of threads is often conflated with a particular implementation, kernel threads. Kernel threads are very low-level abstractions that are provided by the operating system. The nice thing about kernel threads is that they can use any CPU that is the kernel knows about. That’s an important factor in today’s computing landscape, where Moore’s law seems to be giving us more cores instead of more gigahertz.

However, as a building block for a highly concurrent system, kernel threads have a few important problems.

One is that kernel threads simply aren’t designed to be allocated in huge numbers, and instead are more optimized to run in a one-per-CPU-core fashion. Their memory usage is relatively high for what should be a lightweight abstraction: some 10 kilobytes at least and often some megabytes, in the form of the thread’s stack. There are ongoing efforts to reduce this for some systems but we cannot expect wide deployment in the next 5 years, if ever. Even in the best case, a hundred thousand kernel threads will take at least a gigabyte of memory, which seems a bit excessive for book-keeping overhead.

Kernel threads can be a bit irritating to schedule, too: when one thread suspends, it’s for a reason, and it can be that user-space knows a good next thread that should run. However because kernel threads are scheduled in the kernel, it’s rarely possible for the kernel to make informed decisions. There are some “user-mode scheduling” facilities that are in development for some systems, but again only for some systems.

The other significant problem is that building non-crashy systems on top of kernel threads is hard to do, not to mention “correct” systems. It’s an embarrassing situation. For one thing, the low-level synchronization primitives that are typically provided with kernel threads, mutexes and condition variables, are not composable. Also, as with callback-oriented concurrency, one thread can silently corrupt another via unstructured mutation of shared state. It’s worse with kernel threads, though: a kernel thread can be interrupted at any point, not just at I/O. And though callback-oriented systems can theoretically operate on multiple CPUs at once, in practice they don’t. This restriction is sometimes touted as a benefit by proponents of callback-oriented systems, because in such a system, the callback invocations have a single, sequential order. With multiple CPUs, this is not the case, as multiple threads can run at the same time, in parallel.

Kernel threads can work. The Java virtual machine does at least manage to prevent low-level memory corruption and to do so with high performance, but still, even Java-based systems that aim for maximum concurrency avoid using a thread per connection because threads use too much memory.

In this context it’s no wonder that there’s a third strain of concurrency: shared-nothing message-passing systems like Erlang. Erlang isolates each thread (called processes in the Erlang world), giving each it its own heap and “mailbox”. Processes can spawn other processes, and the concurrency primitive is message-passing. A process that tries receive a message from an empty mailbox will “block”, from its perspective. In the meantime the system will run other processes. Message sends never block, oddly; instead, sending to a process with many messages pending makes it more likely that Erlang will pre-empt the sending process. It’s a strange tradeoff, but it makes sense when you realize that Erlang was designed for network transparency: the same message send/receive interface can be used to send messages to processes on remote machines as well.

No network is truly transparent, however. At the most basic level, the performance of network sends should be much slower than local sends. Whereas a message sent to a remote process has to be written out byte-by-byte over the network, there is no need to copy immutable data within the same address space. The complexity of a remote message send is O(n) in the size of the message, whereas a local immutable send is O(1). This suggests that hiding the different complexities behind one operator is the wrong thing to do. And indeed, given byte read and write operators over sockets, it’s possible to implement remote message send and receive as a process that serializes and parses messages between a channel and a byte sink or source. In this way we get cheap local channels, and network shims are under the programmer’s control. This is the approach that the Go language takes, and is the one we use in Fibers.

Structuring a concurrent program as separate threads that communicate over channels is an old idea that goes back to Tony Hoare’s work on “Communicating Sequential Processes” (CSP). CSP is an elegant tower of mathematical abstraction whose layers form a pattern language for building concurrent systems that you can still reason about. Interestingly, it does so without any concept of time at all, instead representing a thread’s behavior as a trace of instantaneous events. Threads themselves are like functions that unfold over the possible events to produce the actual event trace seen at run-time.

This view of events as instantaneous happenings extends to communication as well. In CSP, one communication between two threads is modelled as an instantaneous event, partitioning the traces of the two threads into “before” and “after” segments.

Practically speaking, this has ramifications in the Go language, which was heavily inspired by CSP. You might think that a channel is just a an asynchronous queue that blocks when writing to a full queue, or when reading from an empty queue. That’s a bit closer to the Erlang conception of how things should work, though as we mentioned, Erlang simply slows down writes to full mailboxes rather than blocking them entirely. However, that’s not what Go and other systems in the CSP family do; sending a message on a channel will block until there is a receiver available, and vice versa. The threads are said to “rendezvous” at the event.

Unbuffered channels have the interesting property that you can select between sending a message on channel a or channel b, and in the end only one message will be sent; nothing happens until there is a receiver ready to take the message. In this way messages are really owned by threads and never by the channels themselves. You can of course add buffering if you like, simply by making a thread that waits on either sends or receives on a channel, and which buffers sends and makes them available to receives. It’s also possible to add explicit support for buffered channels, as Go, core.async, and many other systems do, which can reduce the number of context switches as there is no explicit buffer thread.

Whether to buffer or not to buffer is a tricky choice. It’s possible to implement singly-buffered channels in a system like Erlang via an explicit send/acknowlege protocol, though it seems difficult to implement completely unbuffered channels. As we mentioned, it’s possible to add buffering to an unbuffered system by the introduction of explicit buffer threads. In the end though in Fibers we follow CSP’s lead so that we can implement the nice select behavior that we mentioned above.

As a final point, select is OK but is not a great language abstraction. Say you call a function and it returns some kind of asynchronous result which you then have to select on. It could return this result as a channel, and that would be fine: you can add that channel to the other channels in your select set and you are good. However, what if what the function does is receive a message on a channel, then do something with the message? In that case the function should return a channel, plus a continuation (as a closure or something). If select results in a message being received over that channel, then we call the continuation on the message. Fine. But, what if the function itself wanted to select over some channels? It could return multiple channels and continuations, but that becomes unwieldy.

What we need is an abstraction over asynchronous operations, and that is the main idea of a CSP-derived system called “Concurrent ML” (CML). Originally implemented as a library on top of Standard ML of New Jersey by John Reppy, CML provides this abstraction, which in Fibers is called an operation1. Calling send-operation on a channel returns an operation, which is just a value. Operations are like closures in a way; a closure wraps up code in its environment, which can be later called many times or not at all. Operations likewise can be performed2 many times or not at all; performing an operation is like calling a function. The interesting part is that you can compose operations via the wrap-operation and choice-operation combinators. The former lets you bundle up an operation and a continuation. The latter lets you construct an operation that chooses over a number of operations. Calling perform-operation on a choice operation will perform one and only one of the choices. Performing an operation will call its wrap-operation continuation on the resulting values.

While it’s possible to implement Concurrent ML in terms of Go’s channels and baked-in select statement, it’s more expressive to do it the other way around, as that also lets us implement other operations types besides channel send and receive, for example timeouts and condition variables.


1 CML uses the term event, but I find this to be a confusing name. In this isolated article my terminology probably looks confusing, but in the context of the library I think it can be OK. The jury is out though.

2 In CML, synchronized.

* * *

Well, that's my limited understanding of the crushing weight of history. Note that part of this article is now in the Fibers manual.

Thanks very much to Matthew Flatt, Matthias Felleisen, and Michael Sperber for pushing me towards CML. In the beginning I thought its benefits were small and complication large, but now I see it as being the reverse. Happy hacking :)

132 responses

  1. gasche says:

    The multicore-OCaml people have been asking the same sort of questions in the last few years, trying to find a reasonable compromise between generality and cost of implementation. They have a wiki page at https://ocaml.io/w/Multicore.

    One interesting thing about their current design (among many others) is that they propose to let users implement their own scheduler in (OCaml) user-space, instead of implementing a fixed scheduler in the runtime. For this they propose the addition to OCaml of "effects handlers", which could be described shortly as a structured (and typed) style for delimited control.

    Fibers is also a "user-space" implementation of concurrency as it is a Guile library, but you do not seem to emphasize the possibility of implementing domain-specific schedulers -- it is not clear by quickly looking at the code for a non-Scheme expert to see whether the design in fact bakes in a specific scheduler implementation. Is this part of your design considerations?

    (Another recent part of the multicore-OCaml work is the choice to implement a [Reagents](https://github.com/ocamllabs/reagents) library for lockfree concurrency. Reagents, introduced in [this 2012 paper by Aaron Turon](http://www.mpi-sws.org/%7Eturon/reagents.pdf, allow composable construction of lock-free algorithms and data structure. They are more low-level than CML-style primitives (and less expressive: they remain within the subset of algorithms expressible with only atomic locking), but seem to elegantly pander to some performance-justified needs.

  2. wingo says:

    Thanks for stopping by, gasche, and thanks for the links :)

    Regarding scheduling, I don't know what to do. One of the reasons I am working on Fibers as a library is that I know I'm going to make a bunch of mistakes and I don't want to yoke Guile users (and maintainers!) to those mistakes for forever. So your question has a trivial answer in that you can just use some other threading library, or fork this one.

    However that's not very satisfactory :) I will look at what OCaml is doing for inspiration. In the beginning it will probably be a simple work-stealing implementation with no priorities.

    The reagents library looks very interesting, thanks! Incidentally in Fibers is lock-free as well, but using a naive approach -- a mixture of some data structures that can only be manipulated by particular threads, and some that are compare-and-swap over persistent data structures. CAS and GC are so nice together.

    Fibers is currently not pre-emptive. I want to change that at some point, though probably as an option -- some people won't want pre-emption I guess.

  3. Ole Laursen says:

    Have you read this book?

    https://www.amazon.com/Concurrent-Programming-Java%C2%99-Principles-Pattern/dp/0201310090

    It's a bit of an eye opener.

    I'll be reading up on CML, but I think you might be making a mistake here by confounding the need to handle multiple connections with concurrency. The two things should have nothing to do with each other. You should not need concurrency to talk to multiple endpoints.

    Event-based: I think some of the flak event-based systems are getting is somewhat undeserved - many network applications are event-based in nature. Like a GUI, they're waiting for something to happen. So the problem isn't as such the events, it's more the fact that the underlying system is married to the multiple-endpoints-means-multiple-threads idea so the event handler may block if it needs to do something a bit complicated (like replying) and can't be written sequentially without breaking the event loop.

    In the ideal system, I would write my network service with events where it makes sense, and the handlers for those events would then be able to reply etc. without blocking the whole application. I would not have multiple threads or any concurrency, at all times having only one handler running. Unless I really need the concurrency, e.g. for heavy computations, in which case I'd confine the concurrency to a little isolated part. Or split the load between multiple processes, each having no internal concurrency.

    I think other languages are slowly getting there through the await idea.

  4. Patrick Logan says:

    Linda / tuple spaces / JavaSpaces provides a coordination mechanism which is simple, by value, and expressively able to implement anything from CSP channels to pub/sub and more in a single process or multiple distributed processes, depending on the implementation.

    e.g. see the QIX OS use of "kernel linda"

    https://drive.google.com/file/d/0B0cKsRm-3yprc0VsYzJXeV9VNDQ/view?usp=drivesdk

  5. helmet heroes says:

    Thanks regarding delivering these good write-up

  6. happy wheels 2 says:

    You have got astonishing thing these

  7. gamergossip says:

    high top quality flash games on the

  8. newgamers says:

    Free on the internet games to play in the

  9. Read says:

    As far as I read it, I understood less and less with every word. But thanks for this article and trying to explain. Read here about the type of content that I do understand. But I don't only understand, but enjoy as well.

  10. Grant says:

    Ole, the sort of model you describe (events combined into a single queue) is available on some systems, as they allow explicit use/assignment of event queues and so you entirely decide your own threading model. Where it's not available, it's usually trivial to add a thread or two whose sole purpose is to marshall events for your main thread.

  11. flip diving says:

    - I really wanted to send a small word to say thanks to you for the fantastic points you are writing on this site.

  12. Order Research Paper Online says:

    I saw less and less with each word. Be that as it may, much obliged for this article and attempting to clarify. In the first place it will most likely be a basic work-taking usage without any needs. I would not have various strings or any simultaneous, at all times having just a single handler running.

  13. Happy New Year Images says:

    I wasn't sure that I will read the complete article and post a comment here, but the depth in your writing forces me to read till the very end. In fact I have made a site on New Year 2017 which you can check out to download Happy New Year, Images and Wallpaper absolutely free of cost. Wish you all a happy new year 2017 in advance!!

  14. NEWY says:

    NICE

  15. NEWY says:

    NICE

  16. psoriasis says:

    I truly value the sort of subjects you post here. A debt of gratitude is in order for sharing us an extraordinary data that is really useful.

  17. karan says:
  18. MRINAL PANT says:
  19. download showbox apk file says:

    ShowBox is an awesome app that allows you to watch Movies, TV Shows and Series easily.

  20. befikre says:
  21. christmas 2016 wishes says:
  22. neel patel says:
  23. Happy New Year 2017 says:
  24. YO YO says:
  25. Geneva A Gleason says:
  26. Marlow Chauffeur United Kingdom says:

    I saw less and less with every word. In any case, thankful for this article and endeavoring to clear up. here it's not accessible, it's normally minor to include a string or two whose sole design is to marshall occasions for Marlow Chauffeur United Kingdom your principle string. I would not have different strings or any synchronous, at all circumstances having only a solitary handler running.

  27. dipika mishra says:

    The middle of winter has long been a time of celebration around the world. Centuries before the arrival of the man called Jesus, early Europeans celebrated light and birth in the darkest days of winter. Many peoples rejoiced during the winter solstice, when the worst of the winter was behind them and they could look forward to longer days and extended hours of sunlight.

    In Scandinavia, the Norse celebrated Yule from December 21, the winter solstice, through January. In recognition of the return of the sun, fathers and sons would bring home large logs, which they would set on fire. The people would feast until the log burned out, which could take as many as 12 days. The Norse believed that each spark from the fire represented a new pig or calf that would be born during the coming year.

    The end of December was a perfect time for celebrationFeliz Navidad 2016 Navidad 2016Feliz año nuevo 2017

    Frohe weihnachten 2016Frohe weihnachten Status für FacebookCanciones de Navidad 2016 Villancicos de Feliz Navidad 2016 en inglesCanciones de Feliz Navidad 2016
      Frohes Neues Jahr2017 Frohe weihnachten 2016 weihnachtsgrüße 2016 Frohe weihnachten
      Frohe Weihnachten und Neues Jahr 2017 Neues Jahr 2017frohe Weihnachten 2016
     
     
     Frohe Weihnachtsgrüße 2016 Weihnachtsgrüße 2016Frohe Weihnachtsgrüße
     
     Sprüche zu Weihnachten 2016Frohe Weihnachten Sprüche, Wünsche, GedichteGedichte zu weihnachten 2016Weihnachten 2016 Wünsche

    analyze big data big data search data mining with big databig data database architecturehadoop big data database sql books hadoop big data analyticsdefine big databig data computing concept of big dataanalytics data database sql technology it big data big data examples big data miningsources of big datadata analytics companybig data volumemassive data big data storeopen source big data analytics analytics tools of big databig data structurebusiness data analytics applications for big databest big data databasemanaging big databig data analysis methods what's big data big data 3vexplain big data big data intelligence big data overviewanalytics on big datamassive data analysis big data featuresdatabase of big datause of big data big data productsbig data application developmentbig data and data mininghadoop analytics toolslarge data analytics big data data sourcesbig data researchbig data analytics startupadvanced data analyticsdata management big dataanalytics for big data features of big datadatabase sql books big data analytics methodology
     database sql books database sql pdfdatabase sqldatabase sql and nosqlwhat is data analyticsbig data in securitybig data articleshadoop articleshadoop software downloadhadoop trainingsdatabase sql tutorialsdata analytics booksbig data training in gaziabadbig data training in usbig data courses in goahadoop and big dataapplications of rhow to download big data pdfhow to learn big data and get jobbig data jobs and salary packagessalary packages of big data professionalbig data professionalsbig data professionals jobs
     spanish happinesshow to become spanish translatorspanish newbiespanish loveI love you more in spanish
     love and care in spanisbheat in spanishspanish loversspanish worldhow to speak and write spanishspanish language learn spanish very easily
     download the R tutorials for beginnersdownload SAS tutorials freelearn R and sasBig data engineer salary all over india
     I love you more than anyone in spanishhow to say bad in spanishhow to say you are good in spanishwhat are benfits of spanishspanish songs download sadlearn spanish from the websitespanish learning offline
     
     
    In Rome, where winters were not as harsh as those in the far north, Saturnalia—a holiday in honor of Saturn, the god of agriculture—was celebrated. Beginning in the week leading up to the winter solstice and continuing for a full month, Saturnalia was a hedonistic time, when food and drink were plentiful and the normal Roman social order was turned upside down. For a month, slaves would become masters. Peasants were in command of the city. Business and schools were closed so that everyone could join in the fun.

  28. Credit Card Reviews says:

    Nice Info

  29. best male enhancement pills says:

    Wonderful content

  30. How to Start a Blog says:

    Good info

  31. best gastroenterologist in odisha says:

    nice informations

  32. How to become a Real Estate Agent says:

    Nice Information

  33. marco10 says:

    I found a good Website with telephone numbers for customer service Geico customer service

  34. MBA Assignment Help says:

    I loved the way you discuss the topic great work thanks for the share.

  35. DO My Matlab programming Assignment says:

    This is really a great stuff for sharing. Thanks for sharing.

  36. MBA Dissertation Writing Service says:

    I personally like your post; you have shared good insights and experiences. Keep it up.

  37. Mechanical Engineering Project Help says:

    This was a great and interesting article to read. I have really enjoyed all of this very cool information

  38. SPSS Assignment Help says:

    Such a nice post, keep up the fantastic work

  39. Deep Web says:
  40. essay writing says:

    I personally like your post, you have shared good article. It will help me in great deal.

  41. programming help online says:

    Amazing article thanks or sharing..

  42. tips melancong says:

    Hi to everybody, here everyone is sharing such knowledge, so it’s fastidious to see this site, and I used to visit this blog daily tips melancong

  43. kerja dari rumah says:

    Your work is very good and I appreciate you and hopping for some more informative posts kerja dari rumah

  44. tech startups says:

    You completely match our expectation and the variety of our information. tech startups

  45. melancong says:

    Welcome to the party of my life here you will learn everything about me. melancong

  46. Accounting Dissertation Assistance says:

    I loved the way you discuss the topic great work thanks for the share.

  47. affordable website design pakistan says:

    I'm getting excited about this kind of beneficial information of your stuff in the future

  48. Online MBA Assignment Help says:

    This is really a great stuff for sharing. Thanks for sharing.

  49. Programming Computer Network says:

    I personally like your post; you have shared good insights and experiences. Keep it up.

  50. Sustainable Design Assignment Help says:

    My friend recommended this blog and he was totally right keep up the good work

  51. CRM Writing Help says:

    Such a nice post, keep up the fantastic work

  52. Nursing Original Content Writing says:

    My friend recommended this blog and he was totally right keep up the fantastic work!

  53. anuj says:
  54. ADIL says:
  55. Dakisolatie says:

    I quite like reading an article that can make people think. Dakisolatie

  56. soccer results says:

    Super site! I am Loving it!! Will return once more, Im taking your food likewise, Thanks. soccer results

  57. Stuart Zuhlke says:

    yang dicapainya di bidang ilmu pengetahuan dan teknologi, seni dan budaya, olahraga, dan akademis.ironsteelcenter.com ironsteelcenter.comHarga besi beton Sni Ulir Polos Harga besi beton Sni Ulir PolosHarga besi hollow Harga besi hollowHarga besi cnp Harga besi cnpHarga besi unp Harga besi unpHarga wiremesh Harga wiremeshHarga besi wf Harga besi wfHarga besi h beam Harga besi h beamHarga Plat besi Harga Plat besiHarga pipa besi baja sch 40 sch 80 Harga pipa besi baja sch 40 sch 80Harga besi siku Harga besi sikuHarga Plat kapal besi baja bki krakatau steel Harga Plat kapal besi baja bki krakatau steelHarga bondek Harga bondekHarga baja ringan Harga baja ringanHarga Atap spandek Harga atap spandekHarga stainless steel Harga stainless steeljasa konstruksi jasa konstruksi besi baja jasa konstruksi gudang jasa konstruksi gedung jasa konstruksi undangan pernikahan undangan pernikahan simpleundangan pernikahan online udangan pernikahan pinkundangan pernikahan unik undangan pernikahan onlineundangan pernikahan murah undangan pernikahan islamiundangan pernikahan islami undangan pernikahan murahundangan pernikahan elegan undangan pernikahan artisundangan pernikahan unik dan murah contoh undangan pernikahan
    www.gudangbesibaja.com www.gudangbesibaja.comHarga besi cnp Harga besi cnpHarga besi h beam baja Harga besi h beam bajaHarga Plat besi plat kapal Harga Plat besi plat kapalHarga besi siku Harga besi sikuHarga besi unp Harga besi unpHarga besi wf baja Harga besi wf bajaHarga besi beton Sni Ulir Polos Harga besi beton Sni Ulir PolosHarga besi hollow Harga besi hollowHarga pipa besi baja sch 40 sch 80 Harga pipa besi baja sch 40 sch 80Harga wiremesh Harga wiremeshHarga bondek Harga bondekHarga besi Wf Baja Harga besi Wf Bajajasa konstruksi baja wf jasa konstruksi jembatan jasa konstruksi bangunan jasa konstruksi undangan pernikahan elegan dan murah undangan pernikahan eleganundangan pernikahan simple undangan pernikahan elegan dan murahundangan pernikahan artis undangan pernikahan putihudangan pernikahan pink undangan pernikahan unik dan murahundangan pernikahan putih undangan pernikahan unikContoh undangan pernikahan undangan pernikahan

    harga besi beton sni toko besi baja harga besi bahan bangunanharga pipa stainless steel pipa galvanis medium a besi bjkujual baja wf tabel baja krakatau steel harga besi ulir 16 mmharga stainless steel harga baja profil per kg harga besi 12 sniharga besi ulir harga besi wire mesh harga besi 8 mmdaftar harga pipa galvanis harga besi hollow stainless harga besi beton 10harga besi wf 200 harga baja hollow harga besi 13 ulirbesi kanal c galvanis steel rangka besi betondaftar harga besi beton harga pipa hollow harga besi kgjual wiremesh besi beam sni besi betonsupplier besi profil baja iwf harga besi behel 8mmbesi baja pipa galvanized harga besi beton 10mm snikonstruksi baja wf jual expanded metal harga besi ulir 10daftar harga besi hollow besi wire mesh harga sikuharga wiremesh profil baja h beam harga besi siku 4x4Supplier besi harga beam 200 besi siku hargaharga besi baja harga besi cnp 100 harga pipa besi hitamharga pipa baja jual besi cnp pipa seamlessbesi beton murah harga besi unp 100 daftar harga pipa stainless steelharga kanal c besi kanal c harga harga pipaharga besi stainless harga besi cnp 125 pipa stainlessharga besi per kg besi u galvanisharga plat stainless steel besi c pipa besi galvanisbesi unp harga besi cnp 150 harga besi hollow untuk pagarjual besi wf kanal c pipa besi hitamharga baja h beam daftar harga besi kanal c harga besi hollow 40x40

  58. manu says:
  59. Capstone Project Help says:

    Get the dissertation writing service students look for these days with the prime focus being creating a well researched and lively content on any topic.

  60. Do My Programming Homework says:

    Get the dissertation writing service students look for these days with the prime focus being creating a well researched and lively content on any topic.

  61. Essay Writing Service says:

    Get the dissertation writing service students look for these days with the prime focus being creating a well researched and lively content on any topic.

  62. Economics Assignment says:

    Get the dissertation writing service students look for these days with the prime focus being creating a well researched and lively content on any topic.

  63. HR Assignment Help says:

    Get the dissertation writing service students look for these days with the prime focus being creating a well researched and lively content on any topic.

  64. sociology help says:

    Get the dissertation writing service students look for these days with the prime focus being creating a well researched and lively content on any topic.

  65. Accounting Assignment Help says:

    Get the dissertation writing service students look for these days with the prime focus being creating a well researched and lively content on any topic.

  66. Website Hosting Pakistan says:

    I genuinely appreciated understanding it. Sitting tight for some more incredible articles like this from you in the nearing days

  67. Recovery in Aurora: The Public Schools' Response to the July 2012 Movie Theater Shooting (A) Case Solution says:

    I need to offer a go-ahead for such an educational post....its truly astonishing

  68. HBR Cases Solutions says:

    This is really a great stuff for sharing. Keep it up .Thanks for sharing.

  69. Case Analysis says:

    No doubt! It will be truly hard enrolling! I wish you good fortunes!

  70. Casino Online says:

    Your this site is on legal steroids in UK so you are requested to visit the website for buying only genuine steroids in United Kingdom. I will be very happy to see you out there this nice post here.Casino Online

  71. Devesh Binwal says:

    A great Article... Thanks for this awesome article for sharing with us... Also Check

    Hindi Comics 

    Raj Comics Online

    Amar Chitra Katha

    Anthony Comics

    Chacha Chaudhary Comics

    Chetan Bhagat Novels

    DC Comics

    Diamond Comics

    Doga Comics

    Durjoy Dutta Novels

    Hindi Comics Watch Online

    INDRAJAL COMICS

    Inspector Steel Comics

    King Comics

    Kobi Bheriya Comics

    Love Story Novels

    Manoj Comics

    Marvel Comics

    Multistarrer Comics

    Nagraj Comics

    Novels

    Parmanu Comics

    Pinky Comics

    Raj Comics

    Shakti Comics

    Shaktiman Comics

    Super Commando Dhruv Comics

    Super Indian Comics

    Tamil Novels

    Tiranga Comics

    Happy Valentines Day Happy Valentines Day 2017 Happy Valentine Day Valentine Day Quotes Happy Valentines Day Quotes, Valentine Day Quotes for him Valentines Day Quotes for her Anti Valentines Day Quotes Valentines Day Images Valentines day pictures Valentines Day Images

    Worldfree4u World Free for you World free 4 u Movies Download 300 MB Movies Download HD Movies Download Cartoon Movies Download

  72. dsadasdas says:
  73. Electronics Engineering Assignment Help says:

    This is really a great stuff for sharing. Keep it up .Thanks for sharing. Electronics Engineering Assignment Help

  74. Stateflow in Matlab Matlab Homework Help says:

    Stateflow in Matlab Matlab Homework Help Things are very open and intensely clear explanation of issues. was truly information. Your website is very beneficial.

  75. Nursing Homework Help says:

    Really i appreciate the effort you made to share the knowledge. This is really a great stuff for sharing. Keep it up . Thanks for sharing. Nursing Homework Help

  76. Term Paper Writing says:

    I am pretty sure about this information you shared because its really helpful for everyone. Term Paper Writing

  77. free instagram likes may 2014 says:

    I liked your article and I hope you will have many entries or more. free instagram likes may 2014

  78. Norton Support Number UK 0800-098-8371 Norton Helpline Number UK says:
  79. Valentines Day Quotes says:
  80. Manchun says:
  81. asd says:

    asdsadsadsadsadsads fdfsdf

  82. kerja dari rumah dengan ilancee says:

    I read that Post and got it fine and informative. Please share more like that... kerja dari rumah dengan ilancee

  83. online travel booking sites says:

    Just pure brilliance from you here. I have never expected something less than this from you and you have not disappointed me at all. I suppose you will keep the quality work going on. online travel booking sites

  84. uber malaysia says:

    so happy to find good place to many here in the post, the writing is just great, thanks for the post. uber malaysia

  85. plumbers in Allendale NJ says:

    This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information... plumbers in Allendale NJ

  86. carpet cleaning in yuma az says:

    This is very educational content and written well for a change. It's nice to see that some people still understand how to write a quality post! carpet cleaning in yuma az

  87. DUI lawyer says:

    Thanks for posting this info. I just want to let you know that I just check out your site and I find it very interesting and informative. I can't wait to read lots of your posts. DUI lawyer

  88. how to market a website, what to do after you build a website, my website isn’t getting traffic says:

    it was a wonderful chance to visit this kind of site and I am happy to know. thank you so much for giving us a chance to have this opportunity.. how to market a website, what to do after you build a website, my website isn’t getting traffic

  89. tech news says:

    I can set up my new idea from this post. It gives in depth information. Thanks for this valuable information for all,.. tech news

  90. Dyson V8 Absolute says:

    Very useful post. This is my first time i visit here. I found so many interesting stuff in your blog especially its discussion. Really its great article. Keep it up. Dyson V8 Absolute

  91. best commercial meat grinder says:

    I was surfing net and fortunately came across this site and found very interesting stuff here. Its really fun to read. I enjoyed a lot. Thanks for sharing this wonderful information. best commercial meat grinder

  92. buy anabolic steroids online with credit card says:

    This post is good enough to make somebody understand this amazing thing, and I’m sure everyone will appreciate this interesting things. buy anabolic steroids online with credit card

  93. www.acesss.org/ says:

    This is a great article thanks for sharing this informative information. I will visit your blog regularly for some latest post. I will visit your blog regularly for Some latest post. www.acesss.org/

  94. wace timetable 2016 says:

    Really impressive post. I read it whole and going to share it with my social circules. I enjoyed your article and planning to rewrite it on my own blog. wace timetable 2016

  95. http://buyfblikescheap.com/buy-active-facebook-likes/ says:

    Much thanks for composing such an intriguing article on this point. This has truly made me think and I plan to peruse more http://buyfblikescheap.com/buy-active-facebook-likes/

  96. http://buyfblikescheap.com/buying-real-facebook-likes/ says:

    very interesting post.this is my first time visit here.i found so many interesting stuff in your blog especially its discussion..thanks for the post! http://buyfblikescheap.com/buying-real-facebook-likes/

  97. Mrinal pant says:
  98. BMI CALCULATOR says:

    Thanks for the info! A lot of the right. excellent Articlebmi calculator

  99. R Programming Assignment Help says:

    Those who come to read your article will find lots of helpful and informative tips R Programming Assignment Help

  100. Philip Alderson says:

    The Galaxy S8 launch is shaping up to be one of the biggest announcements of the year, as usual. There is still a little longer to wait, however, as Samsung has pushed its launch back; it will not happen at MWC 2017 as it normally does. To know more about Galaxy S8 Release Date You can visit my site.

  101. http://serrurierparis4.parisserrurier.paris/ says:

    Great job !!

  102. soundcloud to mp3 says:
  103. marathi status says:

    Best Marathi Status Available For You Have A Look.

  104. good day quotes says:

    Here are some articles for encouragement and successful life .

    inspirational and motivational quotes

  105. inspirational and motivational quotes says:

    These are the most informative website for daily dose of inspirational quotes which motivates us to work further to achieve our goalsshort mother daughter quotes good day quotes

  106. happy womens day quotes says:
  107. sehat says:

    niche job i like this toko kopi hijau harga murah

  108. 2017 Holi Festival says:

    Your Site Is Very Good and The Post is Well On Topic, Thanks for Sharing it with us.And you can also see my blog on

    Holi Festival: the Festival and Celebration of ColorsHoli Festival HD Wallpaper

  109. serj says:

    Very good points you wrote here..Great stuff...I think you've made some truly interesting points.Keep up the good work. .eyebrow threading

  110. serj says:

    Good way of telling, good post to take facts regarding my presentation subject matter, which i am going to deliver in my college. scalp psoriasis

  111. serj says:

    This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!. plaque psoriasis

  112. serj says:

    I found your this post while searching for information about blog-related research ... It's a good post .. keep posting and updating information. psoriasis

  113. affordable website pakistan says:

    Pretty helpful material, much thanks for this article

  114. Globalization of Hyatt Place Case Solution says:

    I cherished the way you talk about the point awesome work much obliged for the share Your useful post.

  115. HBS Case Study Solution says:

    Dissertation Guidance Provides quality Online Dissertation Help for students.

  116. Case Study Solution says:

    No doubt! It will be truly hard enrolling! I wish you good fortunes!

  117. Online Case Study Help says:

    This is really great work. Thank you for sharing such a good and useful information here in the blog for students.

  118. HBS case analysis says:

    This site and the resources you provide is really nice keep it up.

  119. HBS Case Study Solution says:

    Good way of telling, good post to take facts regarding my presentation subject matter, which i am going to deliver in my college

  120. iOS Mobile Application Development Service says:

    I genuinely appreciated understanding it. Sitting tight for some more incredible articles like this from you in the nearing days

  121. https://datingbrides.com says:

    Good way of telling, good post to take facts regarding my presentation subject matter, which i am going to deliver in my college

  122. emily1 says:
  123. Kassian says:

    Many thanks, this site is really usefulClick HereClicking HereGo HereGoing Here

  124. Keagan says:
  125. Kayden says:
  126. Peyton says:
  127. Keeley says:
  128. Mrinal pant says:
  129. Mrinal pant says:
  130. Mrinal pant says:
  131. Mrinal pant says:
  132. Super Kamagra USA says:

    I don't generally commentary on blogs but as I see this blog I completely motivated to make one praise that I understand that not enough for the significant context on your writing as one that could understand easily.

    Look at here: https://www.puretablets.com

Leave a Reply