wingolog

correct or inotify: pick one

21 May 2018 2:29 PM (inotify | concurrency | linux | unix | igalia)

Let's say you decide that you'd like to see what some other processes on your system are doing to a subtree of the file system. You don't want to have to change how those processes work -- you just want to see what files those processes create and delete.

One approach would be to just scan the file-system tree periodically, enumerating its contents. But when the file system tree is large and the change rate is low, that's not an optimal thing to do.

Fortunately, Linux provides an API to allow a process to receive notifications on file-system change events, called inotify. So you open up the inotify(7) manual page, and are greeted with this:

With careful programming, an application can use inotify to efficiently monitor and cache the state of a set of filesystem objects. However, robust applications should allow for the fact that bugs in the monitoring logic or races of the kind described below may leave the cache inconsistent with the filesystem state. It is probably wise to do some consistency checking, and rebuild the cache when inconsistencies are detected.

It's not exactly reassuring is it? I mean, "you had one job" and all.

Reading down a bit farther, I thought that with some "careful programming", I could get by. After a day of trying, I am now certain that it is impossible to build a correct recursive directory monitor with inotify, and I am not even sure that "good enough" solutions exist.

pitfall the first: buffer overflow

Fundamentally, inotify races the monitoring process with all other processes on the system. Events are delivered to the monitoring process via a fixed-size buffer that can overflow, and the monitoring process provides no back-pressure on the system's rate of filesystem modifications. With inotify, you have to be ready to lose events.

This I think is probably the easiest limitation to work around. The kernel can let you know when the buffer overflows, and you can tweak the buffer size. Still, it's a first indication that perfect is not possible.

pitfall the second: now you see it, now you don't

This one is the real kicker. Say you get an event that says that a file "frenemies.txt" has been created in the directory "/contacts/". You go to open the file -- but is it still there? By the time you get around to looking for it, it could have been deleted, or renamed, or maybe even created again or replaced! This is a TOCTTOU race, built-in to the inotify API. It is literally impossible to use inotify without this class of error.

The canonical solution to this kind of issue in the kernel is to use file descriptors instead. Instead of or possibly in addition to getting a name with the file change event, you get a descriptor to a (possibly-unlinked) open file, which you would then be responsible for closing. But that's not what inotify does. Oh well!

pitfall the third: race conditions between inotify instances

When you inotify a directory, you get change notifications for just that directory. If you want to get change notifications for subdirectories, you need to open more inotify instances and poll on them all. However now you have N2 problems: as poll and the like return an unordered set of readable file descriptors, each with their own ordering, you no longer have access to a linear order in which changes occurred.

It is impossible to build a recursive directory watcher that definitively says "ok, first /contacts/frenemies.txt was created, then /contacts was renamed to /peeps, ..." because you have no ordering between the different watches. You don't know that there was ever even a time that /contacts/frenemies.txt was an accessible file name; it could have been only ever openable as /peeps/frenemies.txt.

Of course, this is the most basic ordering problem. If you are building a monitoring tool that actually wants to open files -- good luck bubster! It literally cannot be correct. (It might work well enough, of course.)

reflections

As far as I am aware, inotify came out to address the needs of desktop search tools like the belated Beagle (11/10 good pupper just trying to get his pup on). Especially in the days of spinning metal, grovelling over the whole hard-drive was a real non-starter, especially if the search database should to be up-to-date.

But after looking into inotify, I start to see why someone at Google said that desktop search was in some ways harder than web search -- I mean we all struggle to find files on our own machines, even now, 15 years after the whole dnotify/inotify thing started. Part of it is that the given the choice between supporting reliable, fool-proof file system indexes on the one hand, and overclocking the IOPS benchmarks on the other, the kernel gave us inotify. I understand it, but inotify still sucks.

I dunno about you all but whenever I've had to document such an egregious uncorrectable failure mode as any of the ones in the inotify manual, I have rewritten the software instead. In that spirit, I hope that some day we shall send inotify to the pet cemetery, to rest in peace beside Beagle.

22 responses

  1. Anon says:

    There is also fanotify https://lwn.net/Articles/339399/

    I'm not sure it solves any problem for you, but IIRC it was create to overcome problems with inotify

  2. Jos says:

    The state of watching file system events without needing priviliges is indeed lamentable. You've not even mentioned the insane amount of memory needed to keep inotify monitors on all directories: userspace is responsible for mapping the fd in the event back to the full path. So the client has to keep all directory paths in memory.

    An efficient kernel interface for this is really needed. Even with fast SSD, the overhead of doing find is still high.

    find ~ -printf '%T@\t%p\n'|wc -l

    on a home dir with 1.5M files takes 35 seconds on my laptop with fast SSD.

  3. Michael Catanzaro says:

    Michael Kerrisk discusses many of these problems in https://lwn.net/Articles/605128/.

  4. Helge Bahmann says:

    I don't think inotify (or fanotify FWIW) was intended for the purposes you describe ("see what some other processes on your system are doing to a subtree of the file system"). It was intended to allow applications to catch up with the *state* of a fileset (or tree) in an "eventually consistent (!)" fashion. As such, it is not intended to (and cannot) replace active scanning/reading of filesystem state, but to cut down on the rate/scope of when/what to scan. Basically: "no event" -> just sleep. "any event" -> rescan such that we have covered everything potentially affected by last event (worst case whole tree).

    For activity trace with temporal ordering it is more efficient to trace the actors (=processes) instead of the objects (=files) because there are far fewer. No file-based notification API can give you that.

  5. Marc says:

    Yeah I think you're confusing "inotify doesn't work" with "inotify doesn't scale" (to multiple directories, processes, etc.)

  6. ak says:

    Yeah! Wow Its amazing game which name unblocked game junction. Its superb because concept of playing and supporting are fabulous. Its speechless thank god.

  7. Thesis writing service says:

    This is an amazing knowledge about computer processors and file system. Computer is the sign of big growing of technology.

  8. Essay help UK says:

    Presently, I know I can set a cron occupation to do as such consistently, yet is there a Like the minute the document is made or changed, something lifts up on it and Taking a gander at inotify, and checking whether it's something near what I need.

  9. Coursework writers uk says:

    If you can assign space adequate to store the event(s) at the same period that you appeal notification, mistake treatment is easy and clear from a project view.

  10. Margaret Watkins says:

    Hello. Thank you very much for the article. It is very important for me.
    For my work, the most important thing is my laptop. After all on it I write various essays for the site https://writemypaper4me.co/.
    But lately I've noticed that the computer has become very hot. I do not think that the ordinary Libre office on my Linux system loads the system or Chrome.
    So thanks, I will now know what to do next time ...

  11. app promotion services says:

    The iOS platform has maintained its current status no matter rivals, which are intimidating making it extraneous with cheaper options. Even though smartphones are growing in large figures, iPhone remains a regular device since it is connected getting a classy amount of people. For this reason reason, iOS apps are seen to be lucrative if they are launched and marketed inside an effective manner. Really the only factor, that you have to keep in mind is not however the obvious method of attracting users for installing your iOS application. It is a skill, which should be learned. That’s how you're able to learn how to increase iOS application downloads while using the given guides.

  12. 70-347 Exact2pass says:

    Thank you very much for this valuable contribution and informative.

  13. cheap essay writing service uk says:

    In data security and programming, a cushion flood, or support invade, is an oddity where a program, while composing information to a cradle, overwhelms the cradle's limit and overwrites adjoining memory areas.

  14. JAVA ASSIGNMENT HELP says:

    Online assignments help services are run by experts and professionals. They are available 24/7. Students can access these websites and services anytime. They provide full access to every problem and solve the problem in dedicated time. Students can save a lot of time and effort. Taking online services is a wise choice by which students can enroll their self in other activities and utilize their time more effectively.

  15. Andrea says:

    I think you should manually approve or use a stronger CAPTCHA for comments that provide an URI/URL as the last few seem to be link farming spam... ^__^;

  16. hole io says:

    wow! These look perfect!! this is truly an amazing article

  17. Alex Dixon says:

    Great article! Very interesting to read.

  18. write my essay says:

    Losing sleep over unfinished essays but can't afford the assistance of costly essay writers? Wondering whether the online service providers will write my essay cheap in UK? Facing a deadline for finishing the essay? Well, your anxieties end here. We have with us a team of essay writing services for UK students who can solve all your writing issues so that you score high grades in your essay assignment. They will deliver the completed essay to you well before the deadline. With decades of experience in helping students, this team of professional essay typers best in UK.

  19. Assignment Help says:

    Thanks for sharing the information keep on sharing new things.

  20. Speedypaper says:

    Useful post. Thanks!

  21. Assignment Help says:

    This post is not just informative but impressive also. The post is so convincing that it created an urge to choose Assignment Help services. You can email us at Info@Myassignmenthelpau.Com or Phone Number: +61-2-8005-8227

  22. Assignment Help says:

    I loved this post and the way the author has explained the points. I am sure that any student after reading this will be compelled to search for online assignment help.
    I am a freelancing subject matter expert with Assignment Help Australia and My Assignment Help OZ. You can email us your query at

Leave a Reply