What's coming up in imag (1)

This is the first part of a series of Blog posts I want to release every 14 days or so on what is currently brewing in the imag (Pull-Request-)Queue.

I hope I can release one article for this series every 14 days - read: I hope there is enough progress so I can write something on it every 14 days.

Lately there is not that much progress, which mainly is caused by me awaiting review by others. I don’t know why, but it is not happening and pinging them does not help that much.

Lets see what is floating around in the Pull-request-Queue of the repository.

#130 - “Doc mod”

This PR is already over 1 month old. It is meant to document features the imag modules might have. Something like “The calendar should be able to {add, remove, list} events” and so on.

At this moment, there are a whole bunch of sub-tasks for this, one for each module I want to introduce to imag. For the lazy, here is a list with short explanations attached:

  • bookmarks
  • borrow-tracker (if you give something to someone, this tracker can be used to remind you after N days that you should ask to get your stuff back. Should use the calendar and the contacts modules)
  • calendar (via external tools, E.G.: vdirsyncer)
  • contacts (via external tools, E.G.: vdirsyncer)
  • counter (simply for counting things)
  • cue cards (If you want to learn things and you like using cue cards, this module might help you. Attaching wiki entries or external references will be made easy via imag, though main focus is on cue-card organisation)
  • dairy (This is rather simple, right? If the hook system works, this will be encrypted by default)
  • images (via extenal tools, E.G.: git-annex)
  • ledger (via external tools, E.G.: ledger-cli)
  • mails (via extenal tools, E.G.: mutt, offlineimap, msmtp)
  • movies (via extenal tools, E.G.: git-annex)
  • music (via extenal tools, E.G.: git-annex)
  • news (via external tools, E.G.: newsbeuter)
  • notes (simple note-taking tool. Will be used by a lot of other modules to attach notes to your stuff)
  • password (via external tools, E.G.: password-store)
  • personal library (To track references and PDFs and handle bibtex (for example))
  • todo (via external tools, E.G.: taskwarrior)
  • weather (for saving the weather in your store, forecasts and such)
  • wiki

The above are all high-level modules. Some more modules will be low-level modules, which are there to access the store in a module-generic way and which could be used by the user to modify single or multiple store entries, though the high-level modules should do this on their own. These modules are “just” interfaces to the respective libraries.

* link helper (CLI for the linking library) * tag helper (CLI for the tagging library) * git helper (not sure on this one, though) * content-to-website (CLI for the entrymarkup library, markup converter for viewing things in the browser) * store-integrity check (find dead links, etc. Not sure on this one either)

I guess this issue will be there for a long time.

#167 / #171 - Linking

167 is a PR for the documentation which defines how linking should work. 171 is the implementation of the things described in 167.

The PR (171) is ready and waits for review. The linking library tries to ensure that the entries are two-way linked and supports both internal and external linking.

All modules should use this library for linking later on, to ensure the linking works.

#174 - Tagging

174 is the PR for the tagging library. It implements a helper for adding, removing and setting tags in an entry. This was added because tagging is rather essential for a lot of modules (bookmarks, counter, cue cards, images, ledger, mails, movies, music, news, notes, personal library, todo, wiki). Some of these modules might be implemented using external tools (“todo” with taskwarrior for example). These modules might re-implement parts of the tagging mechanism of the tool to support imag. I’m not yet sure how to handle these external tools and discussion is clearly needed, of course.

The PR also includes a helper for building a commandline argument or commandline subcommand for specifying tagging actions. These can be used to build up similar commandline interfaces and parse values of them via the tagging library, so we have a clean UI and a single point of failure for tag handling.

#191 - Markup processing

Here I implement a high-level library which helps converting entry-content via different markup-processors into HTML pages.

This library shall be used to show content from the store to the user in a convenient way by translating content which was written in a markup language into HTML. The user of the library does not know what markup processor is used and she shouldn’t care. The library gets an entry and spits out a chunk of HTML which can be shown to the user than.

#192 - “note” module

This is the first “high level module”. The PR implements the “note” module, though it is currently on hold, as I have to wait for three other PRs to be merged (171, 174, 191 - linking library, tagging library and markup library).

#193 - “counter” module

This is another “high level module” - The “counter” module. It also waits on three more PRs to be merged (171, 174, 191).

#200 - Hooks for the store

Well this is the most complex PR right now. It implements the Hook system for the store. The hook system offers different types of hooks, which are realized via traits, so everyone is able to implement hooks for the store.

These types of hooks are supported:

  • post-create hooks
  • post-delete hooks
  • post-read hooks
  • post-retrieve hooks
  • post-update hooks
  • pre-create hooks
  • pre-delete hooks
  • pre-read hooks
  • pre-retrieve hooks
  • pre-update hooks

Parallel execution of Hooks, although there might be RW-Hooks

The challenge here was to implement the traits for the hook types in a generic way, so each hook is able to either access the data it gets in a read-only wait or aquire write access on the data. This is currenly implemented via an “Accessor” type, which is an enumeration over either a read-only accessor or a read-write accessor (this is not integrated in the PR at the time of writing, as this is highly experimental). With this system in place, the hook system of the store is able to decide whether hooks should be ran in parallel or not. For example, if there are only Read-Access-Hooks in the list of hooks which shall be executed for a, say, create() call, the hook system is able to execute these hooks in parallel.

Not all types of hooks provide this feature, as some of the hooks are not allowed to alter data. For example a pre-read hook only gets the StoreId which shall be read, there is no point in altering the ID on read(). A post-read hook might alter an entry and therefor has to be able to provide an “Accessor” object, which then can be called by the store RO or RW.

Sorting hooks?

Another thing in my queue for this PR is the sorting mechanism for the hooks. One hook might be executed before another or after another. Hooks do not know of eachother - so you cannot compare them. So I had to “invent” a system how each hook can tell the store when it wants to be executed. There are five possible values:

  • First
  • earlier
  • doesn’t matter
  • later
  • Last

As far as I can see, this is not yet optimal and I have to come up with something better. That’s the point why this stuff is not yet included in the PR 200 - it is not yet perfect IMO.

What the hook system may be used for

The hook system is meant to move functionality of the store out of the store library. We argued on having the store version controlled via git. One of us doesn’t want that per default, as “It would pull in git as dependency” (I wouldn’t mind). Having a hook system would enable us to provide version control via git or whatever other tool the user uses, without integrating the functionality in the store.

Encryption of store entries via gnupg would be easy, as the entry gets decrypted on reading into memory and encrypted on writing to disk. The store and the modules wouldn’t even notice.

Consistency checking, integrity checking for linking, tagging or whatever else would be rather easy. All these things could be implemented without altering the core of the store library.

Some of these functionality would still be implemented in the core distribution of imag, though. I would really like to add encryption and version control support as libraries to the distribution and register the hooks for them in the store by default. This does not alter the store library, but the runtime library (which initializes the internal store object). Version control would be enabled by default. I’m not sure whether encryption should be enabled by default, though I doubt anyone would argue against encryption, as all the data which is in the store is rather personal (thus personal information management suite).

More PRs

There are some more PRs floating around in the repository right now. None of them is that large, just little enhancements and nice-to-have things, like colored output for the log, error tracing in color, a ismatch!() macro for less noise in code blocks like this:

if !foo.iter().all(|f| match f { &SomeTypeWithValues(_) => true, _ => false }) {
  // ...

// but simply:

if !foo.iter().all(|f| is_match(f, &SomeTypeWithValues(_))) {
  // ...

Lately merged

Some things were merged in the last five days, though I’d love to see more things merged. But as said, there were no reviews, so I couldn’t merge things.

  • The runtime library removed support for short versions of “–rtp” and “–config”, as the modules migth need “-r” or “-c” (#205)
  • Regex are now compiled once, even if they are called in a loop - lazy_static!() helps with this (#206)
  • clap and regex crate dependencies were updated (#207, #214/#215)
  • Some cleanup was done and unsused imports were removed in libimagentrymarkup (#218)

Random thoughts

I’m currently thinking about the problem on how to integrate external tools without breaking user experence.

For example I use taskwarrior and I really love it. I really want to integrate it into imag, though I still want to be able to use the taskwarrior binary from the commandline as I’m used to. But after altering things in taskwarrior, they should be available in imag as well, without some futher hassle.

As taskwarrior stores its contents in JSON files, I could simply parse them and use this data, though I don’t like that idea that much because if the format changes, I have to adapt my software. Also, linking to this content is rather complex, as taskwarrior seems to store everything in one big file rather than in one-file-one-task. Linking to tasks could become a major pain here.

The same goes for ledger-cli. ledger-cli supports putting each transaction into one seperate file, though the user has to cope with this when using ledger from the commandline. At least as far as I can tell.

None of these tools do export an API. Even if they’d export one, I don’t think they would allow to re-implement the storage backend, so I could build an interface to my store implementation. And of course re-implementing their functionality is really not what I want to do.

taskwarrior has a hook system since 2.4.0 which could fit into my needs. ledger has not.