The last 14 days were really exciting. We merged some awesome things and even more awesome things are in the pipeline, but lets focus on the past two weeks in this third update on imag, the personal information management suite for the terminal.

Hooks everywhere!

First of all, the big news: The hook system got merged.

Holy crap, this was a big one! 950 lines were added and 10 were deleted by this merge, 14 files were changed. Well, I do not want to elaborate on the architecture of the hook system now, this really belongs to the documentation. What I want to do in the next 14 days is implementing a libimagstorestdhook library, which contains several builtin hooks for the store, such as:

• A rather simple “Debug”-Hook, which does nothing but printing debug information about the hook execution.
• A VCS hook which uses git to make the store version-controlled
• A crypto hook which encrypts and decrypts the store on the fly.

For the first iteration, the debug hook is enough. The other hooks are nice to have, though I would only create the stubs for them and then focus on even more important things: modules.

Where are my modules?

Well, I got some really interesting things merged in the last 14 days. Some libraries were merged which really make writing modules a joy. Here is the short list:

• libimagentryfilter got a version filter
• libimagentryfilter got some more rather basic filters added
• libimagentryfilter got the ability to filter by tag
• libimaglink initial codebase was merged
• libimagstore got a function to retrieve all entries for one module
• libimagtag got exec functionality

But what does this mean in detail? Let me elaborate:

libimagentryfilter

We have two new filter types in this library. First, the version filter. It enables to filter out entries by their version field from the header, so you can, for example, select only entries which are in a certain range of versions, if that matters to you when implementing a feature for a module. This filter must prove its usability, of course.

The second filter is a tag filter, so you can select entries by tags. There's no CLI functionality for this, yet. I really want to implement this as well.

Some more basic filter functionality was added. The very core of these changes is a filter which takes a predicate closure, all other basic filters (such as the header-field-is-equal-to-filter and so on) were reimplemented using this core filter type.

libimaglink

Yeah, we finally got this code merged! We have a library for linking entries now. I already elaborated this in the last part, so I won't elaborate on it here

libimagstore

This is also really really nice. We have a Store function to retrieve all entries for one module now. And the best thing is: It is lazy, because it returns an iterator! It uses the glob crate internally, which returns an Iterator itself and we wrap this iterator into a StoreIdIterator. With libimagentryfilter you can simple call filter() on this iterator to reduce the disk IO. How awesome is that?

libimagtag

We have exec functionality. What does this mean? It means that you can pass the commandline objects returned by the clap functions for processing the passed commandline flags and an entry and the exec functionality of the libimagtag does what the CLI tells it to do: If the user specified that there should be the tags “foo” and “bar” added but “baz” should be removed, this function does this for you.

I really want all libraries to be like this, so a developer can compose new modules by simply reusing existing functionality like this.

The CLI gets rather verbose with this, but as Rust wants it: explicit is better than implicit, and we do it just like that.

And what about modules?

We got some interesting module code merged as well. I wrote about the libraries first because the libraries are the things where the big changes happen at this moment, later the module section will come first.

So what have we done?

• imag-counter got an interactive mode
• imag-counter got listing functionality
• imag-link was initialized
• imag-store got ArgGroup
• imag-tag was initialized
• imag-view was initialized

In detail:

imag-counter

Interactive mode. That's a really nice thing in my opinion: The counter binary allows you to call interactive mode and provide a specification where you specify which key is bound to which counter. Then, you can type abc (optionally prefixed by +) to increase the counters bound to a, b and c. To decrease them you can prefix with -.

imag-link

The imag-link binary is a helper to link two entries. Not more and not less. External linking is not supported yet, though.

imag-store

This was nothing special. I merged a enhancement to make use of the ArgGroup feature clap offeres. It prevents the user from specifying two (or more) arguments which are mutually exclusive.

imag-tag

The imag-tag binary is a frontend for the libimagtag. It allows the user to add and remove tags from entries.

imag-view

Finally, the imag-view binary helps you viewing entries.

Bugs everywhere? Bugs nowhere!

So far there haven't been that many bugs! We only saw some minor bugs which resulted in rather simple errors. Nothing to serious, really.

What's in the pipeline

There are some minor things which have to be done:

• The imag-tag binary really should use the new exec functionality of the libimagtag to add/remove tags from entries
• A libimagrt functionality to call the $EDITOR must be merged. A friend of mine implemented this, thank you very much, Kai!
• The imag-view binary is implemented without library. A libimagview shall be created and used to reimplement imag-view. Actually a friend of mine volunteered in doing this. Awesome!

And of course there have to be more modules. By now, imag-counter is the only real-world (and not core- or utility-module) which is available. I'm working on imag-notes right now and I have some promising-looking code floating around for a imag-task binary implemented using “taskwarrior”. I started to write a library for writing taskwarrior hooks in Rust which I want to use to implement imag-task. The taskwarrior hook library is not ready and not published, yet.

After these two, I will start working on imag-diary, maybe imag-bookmark and either imag-contact or imag-calendar depending on what crates are available to do icalendar or Maildir stuff. If neither is available, I will do imag-password using pass (after writing another crate to interface with it, of course).

So, a lot of awesome things are planned. Stay tuned!

tags: #linux #open source #programming #rust #software #tools #imag

This is the second article on what is coming up in imag.

In the last 14 days was not that much noise in the repository, though some interesting things happened anyways.

What happened

In the following sections I will describe the PRs that have been merged in the last 14 days.

In general: There were a lot of documentation improvements and also two rather critical bugs have been fixed.

libimagtag/init

The initial codebase for libimagtag was merged. Finally, we have tagging support in form of a library! libimagtag includes a trait Tagable which is implemented on EntryHeader and Entry, so a user of the library can simply call some_entry.add_tag(some_tag) to add a tag.

The user interface for this library is rather simple, though we have now moved some functionality into a library which grants us compatibility over several modules, as entries from all modules now get tagged in the very same way.

This will also come in handy when filtering for tags, for example.

libimagstore/clean-warnings

Some warnings have been removed from the store compiling step, which results in better code quality. These warnings were nothing serious (as the development of the whole suite is not in a usable state, yet), but they will of course be removed (and turned into errors) before the first release of the suite.

doc/alphabetical-ordering

The documentation on the modules is now in alphabetical order. The non-alphabetical order becomes an issue as the number of modules slowly increases, so I switched it to alphabetical order.

doc/linking

The chapter on how linking between modules should work, was added. Finally we have something we can refer to when implementing the linking library, yay!

Of course, the implementation of the linking library was started as well and is in good progress so far.

libimagrt/config-in-toml

This was a major change. The configuration file was in some ini-like format before and now we simply use TOML. We use TOML already internally for our header format and it turned out to be great (and toml-rs, the rust library for toml is awesome as hell), so why not also using it for our configuration file format?

This removed a dependency for the config crate and we already have the knowledge how to toml-rs crate works, so there is no overhead in learning another API!

doc/add-module-files

This added simply some files for documentation of modules. Nothing too serious here.

doc/libimagutil

This added documentation for the libimagutil, which already has some really nice and useful functionality and will get some more as we make progress, so I started to add the appropriate documentation notes in the document.

libimagutil/ismatch-util

This was an awesome piece of rust macro!

It happens that we often do things like

if match foo { Value::String(_) => true, _ => false } {
    // something
}

in our rust code, when using toml-rs for example. One rather complex thing which came up just today (and is therefor not yet included, but I'll write about the problem here) was

if match foo { Some::Type::A(_) => true, _ => false } ||
    match foo { Some::Type::B(_) => true, _ => false } {
    // something
}

The macro which was introduced by this merge simplifies this noisy mess. The following is equivalent to the upper:

if ismatch!(foo, Some::Type::A(_)) || ismatch!(foo, Some::Type::B(_)) {
    // something
}

It is way more readable, isn't it?

libimagstore/update-glob

The glob crate dependency was updated as the 0.2.10 release had some bugs (mainly the PatternError which did not implement std::error::Error) which got solved in 0.2.11.

Thanks to the author for fast response times on issues and questions!

libimagstore/fix-addimplicitdirectory_creating

This was a bugfix, as the store did not yet create directories inside the store.

I started to write a module for imag and tested it by creating a foo/bar and it failed silently. I checked why and it was because the store library did not create the foo directory inside the store. Fixed that and it worked, so I extracted the patch from my module feature branch and submitted it as own PR.

libimagstore/revert-idinstore-check

This was also a hard one. There was a checking function in the store which tried to validate that a store id is inside the store (on the filesystem).

This function was buggy. It was not only this function, but also the algorithm which created the store ids. It generated the store ids (which are actually just PathBufs) as absolute paths, so foo would become /foo. This was a bug.

The store ids are no longer absolute but relative. The store now adds its own path as prefix for each ID and everything just works now.

What's coming up

Lets have a look what might be merged in the next 14 days.

First of all, there is, of course, the hook system. It is completed, though it is neiter reviewed nor merged, yet. I guess it might get some more tweaks but it will get merged with rather minor fixes instead of big changes. I'm really looking forward to get this into imag!

The second thing is: There are a lot of pending PRs which are for libraries. I want to introduce the linking library, a “notes” library which will be the foundation for the “notes” module, tagging filters in libimagentryfilter and libimagentrymarkup which is a utility to translate entry markup content into HTML (for viewing with a browser). All these libraries (besides libimagentrymarkup) are ready for merging as far as I can see, though there was no review, yet.

I also want to add a store feature so one can move an entry (Store::move() and Store::save_as() will be introduced for this). The interface is ready and submitted as PR, though there is no implementation yet.

There are some modules pending as well: A imag view utility for viewing content (mainly a frontend to libimagentrymarkup, though it will also make use of things from libimagentryfilter and possibly others as well), a imag tag utility to modify tags of arbitrary entries and imag link for linking arbitrary entries. There's also a pending PR for imag counter which will add a simple counter module to the imag suite.

All these modules are waiting for the libraries described above to be merged.

imag counter

The imag counter module will be a rather simple module. It supports creating, deleting and listing counters and of course incrementing, decrementing, setting and re-setting them.

This is planned for the first release.

After that, I want to enhance the functionality of the library and provide several counting mechanisms (and composability so other modules can use the library to use the imag counter functionality in arbitrary ways). I want to be able to count and get a graph which shows on which dates (or times) I did the most incrementing-steps, so the counter needs to save the dates it was incremented.

All these things are rather trivial to implement, though I want to make the interface as generic as possible (and provide some simple things like described above), which makes this task rather non-trivial. That's why this is planned for later and not implemented in the first step.

github pages

Another problem I'm working on is github pages. I want to upload the imag crate documentation to github pages, though it is not that simple as the repository does not contain only one crate but several.

I also want to include the code-coverage statistics later on, though this would be another step and would be done in a seperate PR later on.

Readme update

The README.md file of the repository will get an update in the next few days, so it will list a lot more information on what the repository contains and what to expect from the project.

Random thoughts

Lets see... We have some awesome PRs merged in the last 14 days, there are more awesome PRs in the pipeline and about to be merged... so awesomeness everywhere, isn't it?

Well, yes. I really want to get the libraries merged into the master branch. After that and after the hook system is merged, I can implement modules for imag and I'm also able to implement hooks for imag. And I can develop them in parallel, as they do not depend on each other. This is really important to me, as I do not get blocked from my own PRs!

Let's hope we get this stuff in! Maybe this weekend, as I have a hacking session with some of my friends planned for tomorrow, so lets see... maybe I can report some major progress in 14 days!

tags: #linux #open source #programming #rust #software #tools #imag

This is the first part of a series of Blog posts I want to release every 14 days or so on what is currently brewing in the imag (Pull-Request-)Queue.

I hope I can release one article for this series every 14 days – read: I hope there is enough progress so I can write something on it every 14 days.

Lately there is not that much progress, which mainly is caused by me awaiting review by others. I don't know why, but it is not happening and pinging them does not help that much.

Lets see what is floating around in the Pull-request-Queue of the repository.

#130 – “Doc mod”

This PR is already over 1 month old. It is meant to document features the imag modules might have. Something like “The calendar should be able to {add, remove, list} events” and so on.

At this moment, there are a whole bunch of sub-tasks for this, one for each module I want to introduce to imag. For the lazy, here is a list with short explanations attached:

• bookmarks
• borrow-tracker (if you give something to someone, this tracker can be used to remind you after N days that you should ask to get your stuff back. Should use the calendar and the contacts modules)
• calendar (via external tools, E.G.: vdirsyncer)
• contacts (via external tools, E.G.: vdirsyncer)
• counter (simply for counting things)
• cue cards (If you want to learn things and you like using cue cards, this module might help you. Attaching wiki entries or external references will be made easy via imag, though main focus is on cue-card organisation)
• dairy (This is rather simple, right? If the hook system works, this will be encrypted by default)
• images (via extenal tools, E.G.: git-annex)
• ledger (via external tools, E.G.: ledger-cli)
• mails (via extenal tools, E.G.: mutt, offlineimap, msmtp)
• movies (via extenal tools, E.G.: git-annex)
• music (via extenal tools, E.G.: git-annex)
• news (via external tools, E.G.: newsbeuter)
• notes (simple note-taking tool. Will be used by a lot of other modules to attach notes to your stuff)
• password (via external tools, E.G.: password-store)
• personal library (To track references and PDFs and handle bibtex (for example))
• todo (via external tools, E.G.: taskwarrior)
• weather (for saving the weather in your store, forecasts and such)
• wiki

The above are all high-level modules. Some more modules will be low-level modules, which are there to access the store in a module-generic way and which could be used by the user to modify single or multiple store entries, though the high-level modules should do this on their own. These modules are “just” interfaces to the respective libraries.

* link helper (CLI for the linking library) * tag helper (CLI for the tagging library) * git helper (not sure on this one, though) * content-to-website (CLI for the entrymarkup library, markup converter for viewing things in the browser) * store-integrity check (find dead links, etc. Not sure on this one either)

I guess this issue will be there for a long time.

#167 / #171 – Linking

167 is a PR for the documentation which defines how linking should work. 171 is the implementation of the things described in 167.

The PR (171) is ready and waits for review. The linking library tries to ensure that the entries are two-way linked and supports both internal and external linking.

All modules should use this library for linking later on, to ensure the linking works.

#174 – Tagging

174 is the PR for the tagging library. It implements a helper for adding, removing and setting tags in an entry. This was added because tagging is rather essential for a lot of modules (bookmarks, counter, cue cards, images, ledger, mails, movies, music, news, notes, personal library, todo, wiki). Some of these modules might be implemented using external tools (“todo” with taskwarrior for example). These modules might re-implement parts of the tagging mechanism of the tool to support imag. I'm not yet sure how to handle these external tools and discussion is clearly needed, of course.

The PR also includes a helper for building a commandline argument or commandline subcommand for specifying tagging actions. These can be used to build up similar commandline interfaces and parse values of them via the tagging library, so we have a clean UI and a single point of failure for tag handling.

#191 – Markup processing

Here I implement a high-level library which helps converting entry-content via different markup-processors into HTML pages.

This library shall be used to show content from the store to the user in a convenient way by translating content which was written in a markup language into HTML. The user of the library does not know what markup processor is used and she shouldn't care. The library gets an entry and spits out a chunk of HTML which can be shown to the user than.

#192 – “note” module

This is the first “high level module”. The PR implements the “note” module, though it is currently on hold, as I have to wait for three other PRs to be merged (171, 174, 191 – linking library, tagging library and markup library).

#193 – “counter” module

This is another “high level module” – The “counter” module. It also waits on three more PRs to be merged (171, 174, 191).

#200 – Hooks for the store

Well this is the most complex PR right now. It implements the Hook system for the store. The hook system offers different types of hooks, which are realized via traits, so everyone is able to implement hooks for the store.

These types of hooks are supported:

• post-create hooks
• post-delete hooks
• post-read hooks
• post-retrieve hooks
• post-update hooks
• pre-create hooks
• pre-delete hooks
• pre-read hooks
• pre-retrieve hooks
• pre-update hooks

Parallel execution of Hooks, although there might be RW-Hooks

The challenge here was to implement the traits for the hook types in a generic way, so each hook is able to either access the data it gets in a read-only wait or aquire write access on the data. This is currenly implemented via an “Accessor” type, which is an enumeration over either a read-only accessor or a read-write accessor (this is not integrated in the PR at the time of writing, as this is highly experimental). With this system in place, the hook system of the store is able to decide whether hooks should be ran in parallel or not. For example, if there are only Read-Access-Hooks in the list of hooks which shall be executed for a, say, create() call, the hook system is able to execute these hooks in parallel.

Not all types of hooks provide this feature, as some of the hooks are not allowed to alter data. For example a pre-read hook only gets the StoreId which shall be read, there is no point in altering the ID on read(). A post-read hook might alter an entry and therefor has to be able to provide an “Accessor” object, which then can be called by the store RO or RW.

Sorting hooks?

Another thing in my queue for this PR is the sorting mechanism for the hooks. One hook might be executed before another or after another. Hooks do not know of eachother – so you cannot compare them. So I had to “invent” a system how each hook can tell the store when it wants to be executed. There are five possible values:

• First
• earlier
• doesn't matter
• later
• Last

As far as I can see, this is not yet optimal and I have to come up with something better. That's the point why this stuff is not yet included in the PR 200 – it is not yet perfect IMO.

What the hook system may be used for

The hook system is meant to move functionality of the store out of the store library. We argued on having the store version controlled via git. One of us doesn't want that per default, as “It would pull in git as dependency” (I wouldn't mind). Having a hook system would enable us to provide version control via git or whatever other tool the user uses, without integrating the functionality in the store.

Encryption of store entries via gnupg would be easy, as the entry gets decrypted on reading into memory and encrypted on writing to disk. The store and the modules wouldn't even notice.

Consistency checking, integrity checking for linking, tagging or whatever else would be rather easy. All these things could be implemented without altering the core of the store library.

Some of these functionality would still be implemented in the core distribution of imag, though. I would really like to add encryption and version control support as libraries to the distribution and register the hooks for them in the store by default. This does not alter the store library, but the runtime library (which initializes the internal store object). Version control would be enabled by default. I'm not sure whether encryption should be enabled by default, though I doubt anyone would argue against encryption, as all the data which is in the store is rather personal (thus personal information management suite).

More PRs

There are some more PRs floating around in the repository right now. None of them is that large, just little enhancements and nice-to-have things, like colored output for the log, error tracing in color, a ismatch!() macro for less noise in code blocks like this:

if !foo.iter().all(|f| match f { &SomeTypeWithValues(_) => true, _ => false }) {
  // ...
}

// but simply:

if !foo.iter().all(|f| is_match(f, &SomeTypeWithValues(_))) {
  // ...
}

Lately merged

Some things were merged in the last five days, though I'd love to see more things merged. But as said, there were no reviews, so I couldn't merge things.

• The runtime library removed support for short versions of “—rtp” and “—config”, as the modules migth need “-r” or “-c” (#205)
• Regex are now compiled once, even if they are called in a loop - lazy_static!() helps with this (#206)
• clap and regex crate dependencies were updated (#207, #214/#215)
• Some cleanup was done and unsused imports were removed in libimagentrymarkup (#218)

Random thoughts

I'm currently thinking about the problem on how to integrate external tools without breaking user experence.

For example I use taskwarrior and I really love it. I really want to integrate it into imag, though I still want to be able to use the taskwarrior binary from the commandline as I'm used to. But after altering things in taskwarrior, they should be available in imag as well, without some futher hassle.

As taskwarrior stores its contents in JSON files, I could simply parse them and use this data, though I don't like that idea that much because if the format changes, I have to adapt my software. Also, linking to this content is rather complex, as taskwarrior seems to store everything in one big file rather than in one-file-one-task. Linking to tasks could become a major pain here.

The same goes for ledger-cli. ledger-cli supports putting each transaction into one seperate file, though the user has to cope with this when using ledger from the commandline. At least as far as I can tell.

None of these tools do export an API. Even if they'd export one, I don't think they would allow to re-implement the storage backend, so I could build an interface to my store implementation. And of course re-implementing their functionality is really not what I want to do.

taskwarrior has a hook system since 2.4.0 which could fit into my needs. ledger has not.

tags: #linux #open source #programming #rust #software #tools #imag

“imag-store” was merged today! This is the first commandline tool for imag – the commandline personal information management suite!

Okay, well, it is just the bare interface to the store and has no special features, as it is meant for debugging purposes only. Though, it is the first binary and we can finally see some progress at the CLI front! Yay!

What's next?

I'm writing some libraries at the moment, one for tagging entries and one as convenience layer for linking entries to internal and external resources (but this isn't fully specified yet). I'm also writing some utilities for entry headers, so you can query them from the CLI with a DSL (as well as in a module, but the query is interpreted at runtime and isn't build at compiletime which is way to complicated for me at this moment). I'm also trying to get the git layer in the store working, but we have a discussion about whether it makes sense to implement this non-optional or not. I'm strongly for integrating it, as it makes the store code way simpler and of course it is a big point of the store to have version information available.

And modules?

Nothing keeps you from writing one. Really, all the things are already there and implement. One last thing you might want to wait for is the implementation of the retrieve_copy() method of the store which is not completed yet. But all the other things are there and usable (tests are there, but I really would be careful – I do not consider the store “stable” yet)!

So what are the concrete plans?

1. Get the libraries implemented
2. Test the libraries
3. Start writing modules, the first modules will be the “notes” module and the “bookmarks” module, I guess.
4. More modules
5. More modules
6. More modules

tags: #linux #open source #programming #rust #software #tools #imag

I did it! Just hours ago I released imag in version 0.2.0.

Read here why I did that, despite there are few things working and I do not even consider this stable or even stableish.

First of all: This is not production-ready software. Use at your own risk! This is a release for people to notice that some things work and one could start to play around with it. Do not trust imag with your personal data if you do not have backups. There are bugs. This is not perfect. This is alpha quality or pre-alpha quality software!

imag is a personal information management suite for the commandline. Its target audience are commandline- and power-users. It does not reimplement personal information management (PIM) aspects, but re-uses existing tools and standards to be an addition to an existing workflow, so one does not have to learn a new tool before beeing productive again. Some simple PIM aspects are implemented as imag modules, though. It gives the user the power to connect data from different existing tools and add meta-information to these connections, so one can do data-mining on PIM data.

How the release was done

What did I do to get to the release? Well, not that much. I checked out the master branch and created a v0.2.0 branch on it, then started to fix things up for the release. That included:

• Moving the dependencies for all crates to “0.2.0”. That means that all crates in the imag code repository do not depend on other crates from the imag repository by their path but via crates.io and the “0.2.0” version of the crate. I had to do that to be able to publish these crates. This change wasn't done in master, so we can continue to work on multiple crates at the same time and do not have to cargo publish each crate independendently before beeing able to use it in other imag crates.
• I fixed the dependency markup in imag-store (actually two times) and the version information in libimagbookmark and imag-bookmark (in two seperate commits). Why is that? Well... because nobody is perfect. I simply missed them when updating the version strings from “0.1.0” (which is generated by cargo new) to “0.2.0”.

Then I manually published libimagutil and libimagerror because they do not depend on other imag crates themselves. After that I cargo published all crates in a loop until all of them were published. In the end I cargo published bin (the imag crate itself) and .imag-documentation (the documentation crate which does nothing by itself and just depends on all other crates for building the documentation).

I removed the v0.2.0 branch and created an annotated tag for the release.

Why now

I released imag 0.2.0 today because there are only few things left in the milestone for the 0.2.0 release and all whats left is tagged with release/optional, saying that these things are optional for the release (the issues will move to the next milestone, so they might not be there anymore when you click on this link).

The commit I based the release branch on was more-or-less random. It does not mark any special point in time or something, I just started to branch off of it.

Why releasing if not ready?

Why did I do the release if the software is not ready and nowhere from stable or even stableish?

That answer is pretty easy: To have a point to refer to and to get into a cycle (a release cycle). I'm pretty confident that this version is a point where interested people (read: interested developers and possible contributors) could start playing around with the software. Clearly, bugs are there and things will break. But, as imag is a rather complex piece of software and a large project, I wanted to have a point where people can be notified “Hey, this is working(ish) and you could try it out” so they can jump on the train.

And that's really what I want. I do not want to develop this monstrous project alone. I want the collaborate on this and hear what people want and need. This project solves my personal problems with PIM, but I'm sure that others will benefit from it as well and therefor community efforts are deeply encouraged by me. Or at least I hope that I do communicate this clear enough.

What's in there

Do not use imag if you do not have backups. See above for more detailed warning note.

What we have by now:

• imag-bookmark – A tool to keep and organize your bookmarks
• imag-counter – A simple counter tool. Create, delete, increment and decrement counters
• imag-diary – A diary
• imag-link – Link imag entries with other imag entries
• imag-notes – Create notes
• imag-ref – Reference files on your Filesystem with imag
• imag-store – Core plumbing tool. Do not use at all if you do not know what this is for.
• imag-tag – Tag imag entries
• imag-todo – Todo management with imag, using taskwarrior as backend. This is not a replacement for taskwarrior, but can be used to create entries in the imag store for each taskwarrior task and then these entries can be linked with imag-link.
• imag-view – View entries from the imag store
• imag – The imag binary itself. The purpose of this is to be able to call imag view instead of imag-view.

Clearly this is not everything. We have a huge number of libraries which work under the hood of these executables, but users might not care about this.

What's coming

Well, what's coming for the next release? The 0.3.0 milestone already exists and there are a lot of things already registered in it.

When will I be there? Well, my initial goal was to release 0.2.0 before 01-01-2017, so I clearly managed to get it out in time. I also announced that I want to do 3-month release cycles, so the next release would be due by March 31, 2017. I will keep that goal as I'm really not sure how fast I will be. Maybe we can release 0.3.0 early, but as normal in open source: The next version will be there if its ready.

I hope to land a imag-mail module in the next few weeks, so one can start tracking email with imag. I also hope to land imag-annotation to be able to annotate other entries with notes. imag mv will also be a thing that should go into 0.3.0 as well as more imag-view functionality.

And, of course, there will be a ton of enhancements to the libraries in imag.

Can I use imag libraries in my project?

No.

Well, you can, but I do not guarantee any stability or API consistency or something. But if you want, you clearly can use one of the imag libraries (they are on crates.io, so why not). Just keep in mind that interfaces will change and things will break and bugs will happen. We are 0.2.0 here, not 1.x.y. Keep that in mind.

For the curious, non-developers: imag consists not of one command/binary but of many. Each of those has a library accociated with it: libimagdiary and imag-diary where the imag-diary project is only a commandline interface to libimagdiary. There are some general purpose libraries in the imag codebase – for example libimagerror which are used by all other crates for a certain thing, error handling in this case. These libraries can be used independendently of imag itself. As said above, I do not guarantee anything in regard of API stability or such, so one could use them outside of imag, but I do not encourage it.

What now?

Well, I won't change my day-to-day rhythm when developing imag. As said, the 0.2.0 release was done to get some attention on imag and to people notice that some things might work, to play around and to report issues. 0.3.0 will be the same kind of release, but with more features. I don't know when I will be confident and say “Yep, one could now start to actually use this”, but I guess that it won't be 0.5.0 or 0.6.0 (which are not even planned by now). Maybe it will be 0.10.0 – who knows.

It also really depends on whether there will be (long term) contributors or not. If some people start working on imag more frequently and I can get a small community build up around the project, imag might be in a usable state “pretty soon” (read: within 2017 or 2018).

If not, ... I don't know. It's really that kind of project that runs forever. Does that bother me? A bit, ... but working on imag is fun and a pretty nice hobby... so I don't care that much.

That beeing said... feel free to play around by installing the imag crates (and having backups) and maybe you will be in the git log of the next release?

tags: #linux #open source #programming #rust #software #tools #imag

I wrote about imag before. I wrote about the problems I see with PIM on the commandline and how I want to solve them.

Well, it happens that some of my friends stepped up and reviewed the implementation (in manner of Software architecture) and suggested a design-overhaul. So this is what is happening right now.

As I stated before, I wanted the whole project to be a bit like git: each subcommand is an own executable, you can easily re-use parts of the code in your own projects as library and so on. This was one major point why we removed the whole codebase and started from scratch. Well not really from scratch, as the history is preserved and we can re-use parts of it by digging into the git-history. And, of course, the experience is not lost at all.

But before we did that, I set up a document where we can specify and document what we do. The major point of the paper is to define how the modules should work and what kind of interface they should export.

Some of these things are work in progress, some other things are already defined.

The roadmap changed a lot, of course. As we have now several libraries to implement and later glue together, we have a lot work to do on defining interfaces and (re-)implementing functionality.

I consider this as great opportunity to learn a whole bunch about software architecture.

We also applied some tweaks to the repository. We have now homu for merging and editorconfig so we are all in the same coding style.

The paper is written in pandoc-Markdown, so the paper can be read in either Markdown with your favourite text editor, PDF or as web-page (HTML). Both of the latter have to be compiled first, of course.

I hope enough people care about this project, so we can make it happen as fast as possible. I really want this to succeed, as I would personally benefit a lot from it.

More posts on this topic will follow, of course.

tags: #linux #open source #rust #software #tools #imag

In my blog article On the sad state of PIM for nerds I ranted over the lack of a PIM (Personal Information Management) suite for the terminal (and therefor for nerds like me).

I said that such a suite would be awesome to have and I listed some core features such a suite should have. I also said that I might not have the time for this. I was wrong. I just released version 0.0.1 of “imag” today.

“imag” stands for “Information ManaGement”. Not that creative, though I like this name. I just released version 0.0.1, which is in pre-alpha shape. I also only contains one “module” of the whole idea – a bookmarking module, where you can store your bookmarks and add tags to each of them.

The roadmap

There is a roadmap on github, this is the idea:

• Release v0.0.2 : Notes Module.
• Release v0.0.3 : BM + Notes combination, where one can link to bookmarks from a note
• Release v0.0.4 : Show meta-module for showing content of arbitrary modules
• Release v0.0.5 : Todo Module. Might wrap taskwarrior here. Not decided yet.
• Release v0.0.6 : Calendar Module, using icalendar standard
• Release v0.0.7 : Contacts Module, using vcard files

These are the releases planned so far, more planned Modules are (in alphabetical order):

• (Shopping)Lists Module
• Bibliography-Management Module
• Diary
• Ledger – to track personal finance
• Mail Module – to index email and link to from other content
• Movie library tracker – to index movies and link to from other content.
• Music library tracker – to index music and link to from other content.
• News Module – RSS/Atom, maybe using external tools like newsbeuter, undecided.
• Podcast module – implementation details completely undecided here
• Project planner – implementation details completely undecided here
• Wiki Module – expansion of the notes module with subpages and so on

Other things to do

There are more things planned in the github issues, like output formatter, where we can print all things with JSON (for example) so one can include “imag” in scripts. Also, an integrated webserver where one can view content as website would be nice (though I don't like the idea of editing content via a web page). Basically I'm open to all ideas.

Also one major paint point right now is the following: I have not yet integrated git support. So the file store is not yet version controlled, and I really have to do this soon, so I'm save.

I also did not think about encryption yet. I am thinking about encrypting the store and decrypting it on every call to “imag”, but I'm not sure whether this is the right way, as some application might call “imag” in background later (see next chapter). I'm not even sure whether I should encrypt the store at all, as the user should encrypt his or her device anyways.

Implementation, License, call for contributors

Well, as I want to be safe, I implement “imag” in Rust, my new favourite programming language. If you want to learn Rust, you are welcome to join me, as I'm also still learning things.

“imag” is licensed under the terms of LGPL v2. This is because “imag” might be used to be integrated into other software, maybe someone wants to write a curses frontend for “imag” at some point? Maybe vim plugins? Or something similar? I'd welcome it!

That said, I have to implement the core features first. Linking between content is one of my desired core features and will follow as soon as the notes module is implemented. As the readme file also says, there is a lot to implement, but not that much modules with actual user input. Only four modules: Notes, Shoppinglist, though I guess I will implement this one without user-input support as it only holds a list, then the wiki module and the Bibliography module, whereas I'm not sure whether this one has user input as well. So basically two modules have actual user generated input (speaking of text a user is supposed to edit with an text editor). All other modules contain structured data (JSON) the user can edit via the commandline.

All this is not carved in stone yet and I'm happy to argue about the points I just mentioned. Also I'm happy to argue about more modules and of course I'd love to see pull requests on github!

So feel free to contribute!

tags: #linux #open source #rust #software #tools #imag

Almost all issues for the 0.2.0 release of imag are done.

Here are some notes how I want to do releases before the 1.0.0 version of imag, which, of course, is really not there yet. But I had to think about a decent release strategy for the 0.x.y releases, so here the notes.

imag is a personal information management suite for the commandline. Its target audience are commandline- and power-users. It does not reimplement personal information management (PIM) aspects, but re-uses existing tools and standards to be an addition to an existing workflow, so one does not have to learn a new tool before beeing productive again. Some simple PIM aspects are implemented as imag modules, though. It gives the user the power to connect data from existing tools and add meta-information to these connections, so one can do data-mining on PIM data.

First, all the things I note here apply to releases before 1.x.y, so basically all 0.x.y releases.

I want to do 0.x.0 releases with a bunch of new features every 3 months or something like this. This time range is not fixed, and releases will happen if they are ready, but 3 month is a good first idea, in my opinion.

I won't declare a fixed set of features for every new version, but move around some features as I like, but I try to keep the number of changes decent. For example, the 0.2.0 release has 49 closed and 13 open issues/prs at the time of writing. As I sometimes do more PRs per issue and sometimes just one PR for one issue, or even file PRs without an existing issue in the first place, this number might vary, but I guess about 100 issues/PRs per release should be a maximum.

What I want to do, and what I already do: I mark issues as

• optional if they are optional for the release they are attached to
• soft-required if I would like to get them into the release but wouldn't mind moving them to the next release
• hard-required if they have to be in the release.

I also set a release date. hard-required issues will get my attention first, I try to implement them as soon as possible and get them ready. After that I focus on soft-required things and after that on optional things.

hard-required issues might postpone a release date. soft-required and optional issues do not. But if an issue has to move to the next release, because the release-date is near and the issues are not solved by then, they will automatically advance.

That means, if an issue is optional in 0.2.0, it will get soft-required if it has to move to 0.3.0 and hard-required if it has to move to 0.4.0.

This way, one can easily calculate when a feature will definitively be implemented in the imag codebase.

---

Another thing I try to do: Plan ahead a bit. At the time of writing, the milestones for 0.2.0, 0.3.0 and 0.4.0 exist. I won't create a 0.5.0 milestone before 0.2.0 is done, so I try to plan ahead about two releases, which equals about six months.

---

I will do y-releases (as in 0.x.y) if I encounter a really serious bug. But as we do not guarantee anything right now (see below), I doubt that will ever happen.

---

All the things I wrote above are ideas how I want to do it. I will see how things work out and we'll see whether this is doable for me or not. As I already said somewhere, I'm starting my masters degree this week and I don't know how the workload will be. I really hope I can get things implemented as planned, but I might have to move issues to future milestones if I notice that things are not doable for me in the planned time.

Also, as we are in the 0.x.y release phase, I won't give any guarantees on stability and interfaces an so on.

tags: #linux #open source #programming #rust #software #tools #imag

This is the 24th iteration on what happened in the last four weeks in the imag project, the text based personal information management suite for the commandline.

imag is a personal information management suite for the commandline. Its target audience are commandline- and power-users. It does not reimplement personal information management (PIM) aspects, but re-uses existing tools and standards to be an addition to an existing workflow, so one does not have to learn a new tool before beeing productive again. Some simple PIM aspects are implemented as imag modules, though. It gives the user the power to connect data from different existing tools and add meta-information to these connections, so one can do data-mining on PIM data.

What happened?

In the last four weeks, not that much happened. Mainly, because I was on vacation. Semester ended on Feb 21st, so from there on I did exactly one thing: Enjoy my free time.

Nevertheless, we got some things done in the imag codebase:

• #801 merged the initial support for annotations.
• #897 merged a new feature which gives us the possibility to build imag with the early-panic flag, which causes the store to panic on a failure rather than to throw errors. This is mainly for debugging, not for production use.
• #900 merged improvements for the libimagstore documentation.
• #901 merged some notes on long-term todos which would perish in the github issue tracker.
• #902 merged some improvements on the Ruby API.
• #903 improved the libimagrt documentation.
• #907 updated clap.

Four PRs are not in this list because they are too trivial.

What about Ruby? Well, because of the little progress, nothing happened in regards of the Ruby bindings. But we get there, I promise.

What's coming?

I don't know yet. My 2nd semester for the masters course just began and I really don't know what the workload will be. So... this is all in all a rather sad update, I guess. Progress slowed down a lot, I don't know what will be there in the next four weeks...

We'll see...

tags: #programming #rust #software #tools #imag