encyclopedia snabb and the case of the foreign drivers

24 February 2017 5:37 PM (snabb | igalia | networking | dpdk | drivers | lca | lwn | conway's law)

Peoples of the blogosphere, welcome back to the solipsism! Happy 2017 and all that. Today's missive is about Snabb (formerly Snabb Switch), a high-speed networking project we've been working on at work for some years now.

What's Snabb all about you say? Good question and I have a nice answer for you in video and third-party textual form! This year I managed to make it to in lovely Tasmania. Tasmania is amazing, with wild wombats and pademelons and devils and wallabies and all kinds of things, and they let me talk about Snabb.

You can check that video on the youtube if the link above doesn't work; slides here.

Jonathan Corbet from LWN wrote up the talk in an article here, which besides being flattering is a real windfall as I don't have to write it up myself :)

In that talk I mentioned that Snabb uses its own drivers. We were recently approached by a customer with a simple and honest question: does this really make sense? Is it really a win? Why wouldn't we just use the work that the NIC vendors have already put into their drivers for the Data Plane Development Kit (DPDK)? After all, part of the attraction of a switch to open source is that you will be able to take advantage of the work that others have produced.

Our answer is that while it is indeed possible to use drivers from DPDK, there are costs and benefits on both sides and we think that when we weigh it all up, it makes both technical and economic sense for Snabb to have its own driver implementations. It might sound counterintuitive on the face of things, so I wrote this long article to discuss some perhaps under-appreciated points about the tradeoff.

Technically speaking there are generally two ways you can imagine incorporating DPDK drivers into Snabb:

  1. Bundle a snapshot of the DPDK into Snabb itself.

  2. Somehow make it so that Snabb could (perhaps optionally) compile against a built DPDK SDK.

As part of a software-producing organization that ships solutions based on Snabb, I need to be able to ship a "known thing" to customers. When we ship the lwAFTR, we ship it in source and in binary form. For both of those deliverables, we need to know exactly what code we are shipping. We achieve that by having a minimal set of dependencies in Snabb -- only LuaJIT and three Lua libraries (DynASM, ljsyscall, and pflua) -- and we include those dependencies directly in the source tree. This requirement of ours rules out (2), so the option under consideration is only (1): importing the DPDK (or some part of it) directly into Snabb.

So let's start by looking at Snabb and the DPDK from the top down, comparing some metrics, seeing how we could make this combination.

Code lines 61K 583K
Contributors (all-time) 60 370
Contributors (since Jan 2016) 32 240
Non-merge commits (since Jan 2016) 1.4K 3.2K

These numbers aren't directly comparable, of course; in Snabb our unit of code change is the merge rather than the commit, and in Snabb we include a number of production-ready applications like the lwAFTR and the NFV, but they are fine enough numbers to start with. What seems clear is that the DPDK project is significantly larger than Snabb, so adding it to Snabb would fundamentally change the nature of the Snabb project.

So depending on the DPDK makes it so that suddenly Snabb jumps from being a project that compiles in a minute to being a much more heavy-weight thing. That could be OK if the benefits were high enough and if there weren't other costs, but there are indeed other costs to including the DPDK:

  • Data-plane control. Right now when I ship a product, I can be responsible for the whole data plane: everything that happens on the CPU when packets are being processed. This includes the driver, naturally; it's part of Snabb and if I need to change it or if I need to understand it in some deep way, I can do that. But if I switch to third-party drivers, this is now out of my domain; there's a wall between me and something that running on my CPU. And if there is a performance problem, I now have someone to blame that's not myself! From the customer perspective this is terrible, as you want the responsibility for software to rest in one entity.

  • Impedance-matching development costs. Snabb is written in Lua; the DPDK is written in C. I will have to build a bridge, and keep it up to date as both Snabb and the DPDK evolve. This impedance-matching layer is also another source of bugs; either we make a local impedance matcher in C or we bind everything using LuaJIT's FFI. In the former case, it's a lot of duplicate code, and in the latter we lose compile-time type checking, which is a no-go given that the DPDK can and does change API and ABI.

  • Communication costs. The DPDK development list had 3K messages in January. Keeping up with DPDK development would become necessary, as the DPDK is now in your dataplane, but it costs significant amounts of time.

  • Costs relating to mismatched goals. Snabb tries to win development and run-time speed by searching for simple solutions. The DPDK tries to be a showcase for NIC features from vendors, placing less of a priority on simplicity. This is a very real cost in the form of the way network packets are represented in the DPDK, with support for such features as scatter/gather and indirect buffers. In Snabb we were able to do away with this complexity by having simple linear buffers, and our speed did not suffer; adding the DPDK again would either force us to marshal and unmarshal these buffers into and out of the DPDK's format, or otherwise to reintroduce this particular complexity into Snabb.

  • Abstraction costs. A network function written against the DPDK typically uses at least three abstraction layers: the "EAL" environment abstraction layer, the "PMD" poll-mode driver layer, and often an internal hardware abstraction layer from the network card vendor. (And some of those abstraction layers are actually external dependencies of the DPDK, as with Mellanox's ConnectX-4 drivers!) Any discrepancy between the goals and/or implementation of these layers and the goals of a Snabb network function is a cost in developer time and in run-time. Note that those low-level HAL facilities aren't considered acceptable in upstream Linux kernels, for all of these reasons!

  • Stay-on-the-train costs. The DPDK is big and sometimes its abstractions change. As a minor player just riding the DPDK train, we would have to invest a continuous amount of effort into just staying aboard.

  • Fork costs. The Snabb project has a number of contributors but is really run by Luke Gorrie. Because Snabb is so small and understandable, if Luke decided to stop working on Snabb or take it in a radically different direction, I would feel comfortable continuing to maintain (a fork of) Snabb for as long as is necessary. If the DPDK changed goals for whatever reason, I don't think I would want to continue to maintain a stale fork.

  • Overkill costs. Drivers written against the DPDK have many considerations that simply aren't relevant in a Snabb world: kernel drivers (KNI), special NIC features that we don't use in Snabb (RDMA, offload), non-x86 architectures with different barrier semantics, threads, complicated buffer layouts (chained and indirect), interaction with specific kernel modules (uio-pci-generic / igb-uio / ...), and so on. We don't need all of that, but we would have to bring it along for the ride, and any changes we might want to make would have to take these use cases into account so that other users won't get mad.

So there are lots of costs if we were to try to hop on the DPDK train. But what about the benefits? The goal of relying on the DPDK would be that we "automatically" get drivers, and ultimately that a network function would be driver-agnostic. But this is not necessarily the case. Each driver has its own set of quirks and tuning parameters; in order for a software development team to be able to support a new platform, the team would need to validate the platform, discover the right tuning parameters, and modify the software to configure the platform for good performance. Sadly this is not a trivial amount of work.

Furthermore, using a different vendor's driver isn't always easy. Consider Mellanox's DPDK ConnectX-4 / ConnectX-5 support: the "Quick Start" guide has you first install MLNX_OFED in order to build the DPDK drivers. What is this thing exactly? You go to download the tarball and it's 55 megabytes. What's in it? 30 other tarballs! If you build it somehow from source instead of using the vendor binaries, then what do you get? All that code, running as root, with kernel modules, and implementing systemd/sysvinit services!!! And this is just step one!!!! Worse yet, this enormous amount of code powering a DPDK driver is mostly driver-specific; what we hear from colleagues whose organizations decided to bet on the DPDK is that you don't get to amortize much knowledge or validation when you switch between an Intel and a Mellanox card.

In the end when we ship a solution, it's going to be tested against a specific NIC or set of NICs. Each NIC will add to the validation effort. So if we were to rely on the DPDK's drivers, we would have payed all the costs but we wouldn't save very much in the end.

There is another way. Instead of relying on so much third-party code that it is impossible for any one person to grasp the entirety of a network function, much less be responsible for it, we can build systems small enough to understand. In Snabb we just read the data sheet and write a driver. (Of course we also benefit by looking at DPDK and other open source drivers as well to see how they structure things.) By only including what is needed, Snabb drivers are typically only a thousand or two thousand lines of Lua. With a driver of that size, it's possible for even a small ISV or in-house developer to "own" the entire data plane of whatever network function you need.

Of course Snabb drivers have costs too. What are they? Are customers going to be stuck forever paying for drivers for every new card that comes out? It's a very good question and one that I know is in the minds of many.

Obviously I don't have the whole answer, as my role in this market is a software developer, not an end user. But having talked with other people in the Snabb community, I see it like this: Snabb is still in relatively early days. What we need are about three good drivers. One of them should be for a standard workhorse commodity 10Gbps NIC, which we have in the Intel 82599 driver. That chipset has been out for a while so we probably need to update it to the current commodities being sold. Additionally we need a couple cards that are going to compete in the 100Gbps space. We have the Mellanox ConnectX-4 and presumably ConnectX-5 drivers on the way, but there's room for another one. We've found that it's hard to actually get good performance out of 100Gbps cards, so this is a space in which NIC vendors can differentiate their offerings.

We budget somewhere between 3 and 9 months of developer time to create a completely new Snabb driver. Of course it usually takes less time to develop Snabb support for a NIC that is only incrementally different from others in the same family that already have drivers.

We see this driver development work to be similar to the work needed to validate a new NIC for a network function, with the additional advantage that it gives us up-front knowledge instead of the best-effort testing later in the game that we would get with the DPDK. When you add all the additional costs of riding the DPDK train, we expect that the cost of Snabb-native drivers competes favorably against the cost of relying on third-party DPDK drivers.

In the beginning it's natural that early adopters of Snabb make investments in this base set of Snabb network drivers, as they would to validate a network function on a new platform. However over time as Snabb applications start to be deployed over more ports in the field, network vendors will also see that it's in their interests to have solid Snabb drivers, just as they now see with the Linux kernel and with the DPDK, and given that the investment is relatively low compared to their already existing efforts in Linux and the DPDK, it is quite feasible that we will see the NIC vendors of the world start to value Snabb for the performance that it can squeeze out of their cards.

So in summary, in Snabb we are convinced that writing minimal drivers that are adapted to our needs is an overall win compared to relying on third-party code. It lets us ship solutions that we can feel responsible for: both for their operational characteristics as well as their maintainability over time. Still, we are happy to learn and share with our colleagues all across the open source high-performance networking space, from the DPDK to VPP and beyond.

60 responses

  1. fd says:

    I enjoyed the youtube video of your presentation.

    I presume snabb doesn't copy packets between apps, so I'm wondering how packet buffer memory is managed along an entire chain of apps. Is ownership of a buffer transferred from one app to the next during link receive/transmit operations? Are buffers reference-counted so they can be shared among apps (think fan-outs)? If buffers are reference-counted, are they read-only? If an app desires to modify a buffer, can it obtain exclusive access? Or does it need to make a copy? etc.

  2. Blake says:

    Thanks for keeping the code clean and simple, Andy!

    Hopefully the driver code in VPP will help you accelerate development of new Snabb drivers as well.

  3. Cars 3 says:

    Cars 3 will depict Lightning MacQueen getting into a depression because of all the new generation cars.. And the new antagonist Jackson Storm will be seen taking over Lightning.

  4. windows login windows 10 says:

    It's look like a very good blog post and informative too.I also like to inform all of you to follow the given web address and comes to know why and how to log off windows 10 desktop with short cut keys.Thank you to this valuable post.

  5. soundcloud converter says:

    In the beginning it's natural that early adopters of Snabb make investments in this base set of Snabb network drivers, as they would to validate a network function on a new platform.

  6. home care Tampa says:

    Thank for this well explained information.Expressed in a better way.

  7. KM Couplings says:

    I am beginner here.But very much interested to know about this.Thank you

  8. says:

    get tutorial for how to hide task bar in windows 10 without spending a single penny.

  9. youtube red free apk says:

    Your article is something I found really very helpful. But is it okay if I share the content with my friends over email?

  10. hd cinema apk says:

    great piece of work

  11. soundcloud playlist converter says:

    As usual, great post. You keep amazing me with your content design and layout.

    And thanks for sharing this awesome piece!

  12. Soundcloudify says:

    отличный саундклауд даунлоадер, залетайте, тестируйте, получайте удовольствие

  13. Hummingbird Loans says:

    Be bound to pay the loan back promptly and enough money within your bank account, if your payday loan is outstanding. Quick loans online is the only way to borrow, but it is undoubtedly the fastest way to get extra money on the internet. Biz2Credit provides a Small business Loans, Business Start up loans, SBA loans, business acquisition loans, equipment financing, and franchisee loan to small and medium businesses owners to grow their business.

  14. Custom essay writing services says:

    I am truly thankful to the owner of this web site who has shared this impressive piece of writing at this place.

  15. 2019 says:

    How to easily download images for free

  16. online mba assignment writing says:

    An encyclopedia is a voluminous, brilliant arrangement of books that endeavor to cover all learning - or that go into profundity about a specific claim to fame, for example, the encyclopedia of, state, Greek Mythology. At my home, before the presence of Google, reference books were dismantled out amid suppers to respond to questions and settle family contentions. So as to have the most refreshed data it was important to purchase another set with a certain consistency. The foundation of "encyclopedia" originates from "circle", proposing a balanced training. (This is so lovely I can scarcely stand it.)

  17. Top Coupon Codes says:

    I really happy to read this post,I was just imagine about it and you provided me the correct information.

  18. scdownloader says:

    I am truly thankful to the owner of this web site who has shared this impressive piece of writing at this place.

  19. says:

    nice one

  20. says:

    nice one good post

  21. says:

    super post

  22. says:

    nice one

  23. Rituparna Das says:

    Such a lovely post which are make here and i will follow you on social media.. thanks

  24. Radhika Radha says:

    Shopping Offers, Free Recharge Tricks, Loot Offers, Coupon Codes,
    Amazon Offers, Recharge Offers.
    Free Paytm Cash>>

  25. Nagaland State Lotteries says:

    Tech News, Tips and Trick, Nagaland Lottery Result, Nagaland State Lotteries

  26. Lottery Sambad says:

    Lottery Sambad, Kerala lottery, West Bengal Lottery Result.
    Lottery News Daily

  27. Nagaland Lottery Result says:

    Nagaland Lottery Result, Nagaland State Lotteries Result

  28. Cheap web hosting says:

    Cheap web hosting. Cheap web hosting. Cheap web hosting. Cheap web hosting

  29. happy wheels says:

    I am also a driver, I have many problems but I have learned a lot of things through your article.

  30. tree service says:

    Thanks for the things you share, we always like this to build a wonderful community that is a place to share knowledge, communicate with everyone.

  31. router login says:

    nice post

  32. says:

    Thanks for the things you share

  33. says:

    good one

  34. Sapna Ranaut says:

    Nice post Thank you very Much for this.

  35. kajal models says:

    VIP ESCORT SERVICE IN DEHRADUN CITY High profile college girl Vip model Indian Russian hotel and home service outcall incall service available24hr FOR BEST DEAL IN YOUR SERVICES REQUIRED VALUE FOR MONEY SERVICE ANY STYLE SERVICE AND FULL SATISFIED IN CALL OUT CALL SERVICE AVALIBLE HERE All models AVALIBLE here Service required details You take someone services..

  36. suhani arora says:

    Haridwar Call Girls are private and high class escorts Haridwar with gorgeous looks and intelligence Avails For ?Vip Girls ?Model ?Russian ?Independent etc.

  37. Lottery Sambad says:


    Today I would like to introduce a good site for Sambad Lottery Result. The lottery users can check and free download from

  38. My Assignment help says:

    If you are looking for a professional company you can help you with your need for my assignment help services then, get yourself connected to IdealAssignmentHelp now. We have the most reliable and professional team in the business to help you with top-quality assignment writing services. Our experts are highly experienced and educated enough to deliver the goods for you as per your specific needs and requirements. Get Connected now for Assignment Help!

  39. 192.168.l.l says:

    Great Information sharing.

  40. rupshikarai says:

  41. Aj Agoncillo says:

    Excellent and very exciting site. Love to read you post. Keep Rocking.

  42. 192.168.l.l says:

    No doubt this is an excellent post I got a lot of knowledge after reading good luck. Theme of blog is excellent there is almost everything to read, Brilliant post.

  43. Delhi escorts says:

    very clear review, it shows you have innovative outlook, wonderful blog. Thanks man.

  44. Noida Escorts says:

    If you don't mind ensure you are postung inclining point here. You rock for every one of the stuffs.

  45. Shweta Mahajan says:

    Hello friends, if you want to spend quality time in the company of sweet and sexy Delhi Call Girls, meet me. My name is Shweta Mahajan and I am one of the Independent Delhi Escorts who offers good companionship.

  46. Cassandra D. says:

    Thanks a lot for your time to have had these things together on this blog.

  47. Cassandra D. says:

    Thanks a lot for your time to have had these things together on this blog.

  48. Paharganj Escorts says:

    Time when people attempt to hide their mistakes is worst thing ever, this line is well written on this article. Thank you.

  49. Kolkata Escorts says:

    Welcome to the Kolkata Hot-babes Escórts Agency. Rupshika Rai best kolkata Escórts girls giving just incredible Sensual can fulfill each and every lustful desire without any limitations.

  50. tndte says:

    ihfrbg iwfbwevr iwevr wer w

  51. Property management Perth says:

    Hi! I like this site because its easy access! The webpage is so amazing!

  52. Patios Perth says:

    This website has an attractive webpage that everyone can see it. So nice!

  53. Vape Nz says:

    With the advancement of technology come perks such as the electronic cigarette which is better for you and the environment.

  54. tonosdellamadacanciones says:

    I was in and introduced a place to share quality ringtones.

  55. Wayne Ogilvie says:

    What a great insight about encyclopedia snabb and the case of the foreign drivers. If you need a remodeling project done for a store or any other type of commercial building we can help.

  56. iphone 着信音 says:

    You can try discovering a few cell phone ringtones

  57. 8 ball says:

    Thank you very much for these great cake recipes, I have learned a lot from your web blog

  58. 192.168.l 254 says:

    IP address can be used to change the name of the Wi-Fi, control the number of devices that are connected at a time to the network and much m

  59. 192.168 01 says:

    There are more than 18 million IP addresses available worldwide. As we have talked about this in other posts there are two IP address types that are available

  60. lottery sambad says:

    The Lottery Sambad today result is uploaded in two different HD file formats the PDF and DBF formats. Both are downloadable and also you can view it online on our website. Lottery Sambad Today Result 4pm are also uploaded you can check it by clicking on the above button. Quick Guide to Download Lottery Sambad Result.

Leave a Reply