Another exciting two weeks! This is the fourth iteration on what's coming up in imag, my personal information management suite for the commandline.

And we had an awesome bunch of merges in the last 14 days!

Editors everywhere!

We now have a utility function in the runtime library for calling the $EDITOR, which either uses the editor specified by the commandline arguments, or the configuration, or the $EDITOR variable from the environment.

This was implemented by a friend of mine who wants to learn rust as well. Thanks a lot!

Warnings? No, Errors!

We got two patches merged (one for the store library, one for the runtime library) which turned warnings of a certain kind into hard errors and prevent compilation. I take this as “We get more stable”.

More debug output!

A whole bunch of merges were done to enhance the debug output in a lot of cases. For example, the Store object implements Debug now, which means that debugging the store is way simpler than before.

Bugfixes!

Remember me writing about how few bugs we had by now? Well... we had some but did not find them. We merged a bunch of bugfix-pull-requests in the first part of the last 14 days. Most of them were rather simple bugs (one was even a one-character-was-wrong bugfix) but nevertheless critical.

Readusall!

I added one readme file for each crate in the project, so people can see what are the libraries supposed to do. Awesome, isn't it?

New Issues!

I've added a whole bunch of issues in the last 14 days, and all of them are tagged with either complexity/easy, complexity/medium or complexity/high as far as I can remember. So if you want to have a look what's going on and maybe help with some pull requests, the ideas are already there!

Here's a short iteration of the issues I opened...

• #282 - libimagrt config returns ConfigNotFound even if it was a Parser error. The runtime library returns a RuntimeErrorType::ConfigNotFound even when there was a parser error. This has to be fixed. Complexity: Easy.
• #284 - imag-tag should use libimagtag::exec::exec_cli_for_entry(). The imag-tag module does not yet use the libimagtag functionality for executing the commandline specification via the library and instead reimplements this functionality. Complexity: Easy.
• #294 - Store should automatically fallback to latest version of entry. The libimagstore::store::Store type does only load the entry specified. By using the libimagstore::storeid::StoreId type and the generator macro for generating these objects, we enforce the version of the entry to be present. What we do not do is to allow the version part to be optional (when building the entry path from a commandline specification). If we would allow this, the store would need to fall back onto the latest version of the entry which is currently available. The store does not do this at the moment, but I think this would be a nice feature, actually. Complexity: High.
• #297 - Library for extracting links from content. There has to be a markup library which allows us to extract links from a Markup document such as Markdown, TexTile and so on. This library and libimaglink could then be used to create the header links via some store hooks. Complexity: medium / maybe high.
• #298 - Hook: Verify that all linked entries exist. That's a rather simple one: We need to check whether all links from a header are actually existing files in the store. We want to do this as hook and we want the hook to be configurable to either deny entry changes or just print warnings.
• #299 - Check whether we want ot use Cow in our APIs. Another rather simple one: We want to check whether it would be beneficial if we'd use Cow in our internal APIs. Mostly for String types, but maybe also for other types as well. Complexity: Easy.
• #300 - Rewrite libimaglink – external linking. The external linking library module has to be rewritten. See the issue for more information. Complexity: Medium.
• #301 - module: Bookmarks. We want to introduce the bookmarking module. Complexity: Easy.
• #302 - Store Hook Aspect configuration: Setting do deny mutable hooks. We want to be able to configure hook aspects to deny mutable hooks. This is mainly a security-related task and would be a nice to have. Complexity: Medium.
• #303 - Make store hooks be executed in parallel if possible. We want to alter the hook execution algorithm to execute hooks in parallel if possible. Complexity: High.
• #304 - imag-view: libimagentryview has to be written. We want to extract the viewing functionality from imag-view and add some more functionality to it. This should be put into a new library libimagentryview. Actually there's someone working on this. Complexity: Easy.
• #306 - module imag-stat. Another nice-to-have: A module for getting some stats/diagnostics about the store. Complexity: Medium.
• #307 - libimaglink: XDG-open for external links. We want to add a feature in libimaglink to open external links via xdg-open. Complexity: Easy, can only be started after #300 was merged.
• #308 - Logging: Print file and line on debug!(). We should alter our logger backend implementation to print file and line on each debug!() log call as well. Complexity: Easy. This Issue was closed at the time of publishing this blog article.
• #312 - Rename: libimaglink –> libimagentrylink. We should rename this library for more consisteny. Complexity: Easy.
• #313 - Rename: libimagtag –> libimagentrytag. We should rename this library for more consisteny. Complexity: Easy.
• #314 - Remove warnings from libimagutil. We should remove the build-time warnings from this library. Complexity: Easy.
• #315 - Convert TOML destructuring into toml::Value::lookup(). As the TOML crate supports Value::lookup() to lookup values at a specific path, we can use this functionality to replace a lot of boilerplate code which simply destructs TOML objects. Complexity: Medium, because a lot of code has to be checked and altered. The changes themselves might not be that complex.
• #316 - module: weather. I added this issue for tracking of the implementation of the weather module. Yes, we plan a weather module for imag, see the linked issue what I expect such a module to do. Complexity: Easy.

This list could be seen as a cry for contributors! At least there is now a list available what has to be done and with some complexity level attached - contributors can now see where to start!

Side-Project: task-hookrs

This is a little side-project which I started. It aims to implement a crate for interacting with taskwarrior, so you can export tasks from taskwarrior and import them into your rust program. Of course I'm writing this so I can use it in imag to implement the todo module by using taskwarrior as todo manager and interface with it through taskwarrior hooks and my task-hookrs crate.

In the last 14 days I got some important points working: Deserializing and Serializing are already working and so is importing from a Read. Exporting is on its way.

Some things I didn't implement yet is support for the id and urgency fields and annotations are missing as well, but I guess these are really simple to implement as well.

I'm really looking forward to get this crate working in the next days and I'm starting to wonder whether I should put in on crates.io and how this whole process works.

What's coming up

As always, I want to close this article with a prospect. I got less stuff done than I expected in the last 14 days:

$ git diff --shortstat $(git log --since=14days --format=%H | tail -n 1)
56 files changed, 1193 insertions(+), 86 deletions(-)

I'd love to get some more progress in the next 14 days, though I cannot guarantee it, as I have to write a bachelors thesis and as I have to do some other important things as well. I really hope I can get the diary module ready and merged within the next days and maybe I can start implementing the bookmark manager then (after I solved #300). As this is not that complicated besides the #300 which has to be merged first, I might get started with the todo module as well. I really hope so.

tags: #linux #open source #programming #rust #software #tools #imag

The last 14 days were really exciting. We merged some awesome things and even more awesome things are in the pipeline, but lets focus on the past two weeks in this third update on imag, the personal information management suite for the terminal.

Hooks everywhere!

First of all, the big news: The hook system got merged.

Holy crap, this was a big one! 950 lines were added and 10 were deleted by this merge, 14 files were changed. Well, I do not want to elaborate on the architecture of the hook system now, this really belongs to the documentation. What I want to do in the next 14 days is implementing a libimagstorestdhook library, which contains several builtin hooks for the store, such as:

• A rather simple “Debug”-Hook, which does nothing but printing debug information about the hook execution.
• A VCS hook which uses git to make the store version-controlled
• A crypto hook which encrypts and decrypts the store on the fly.

For the first iteration, the debug hook is enough. The other hooks are nice to have, though I would only create the stubs for them and then focus on even more important things: modules.

Where are my modules?

Well, I got some really interesting things merged in the last 14 days. Some libraries were merged which really make writing modules a joy. Here is the short list:

• libimagentryfilter got a version filter
• libimagentryfilter got some more rather basic filters added
• libimagentryfilter got the ability to filter by tag
• libimaglink initial codebase was merged
• libimagstore got a function to retrieve all entries for one module
• libimagtag got exec functionality

But what does this mean in detail? Let me elaborate:

libimagentryfilter

We have two new filter types in this library. First, the version filter. It enables to filter out entries by their version field from the header, so you can, for example, select only entries which are in a certain range of versions, if that matters to you when implementing a feature for a module. This filter must prove its usability, of course.

The second filter is a tag filter, so you can select entries by tags. There's no CLI functionality for this, yet. I really want to implement this as well.

Some more basic filter functionality was added. The very core of these changes is a filter which takes a predicate closure, all other basic filters (such as the header-field-is-equal-to-filter and so on) were reimplemented using this core filter type.

libimaglink

Yeah, we finally got this code merged! We have a library for linking entries now. I already elaborated this in the last part, so I won't elaborate on it here

libimagstore

This is also really really nice. We have a Store function to retrieve all entries for one module now. And the best thing is: It is lazy, because it returns an iterator! It uses the glob crate internally, which returns an Iterator itself and we wrap this iterator into a StoreIdIterator. With libimagentryfilter you can simple call filter() on this iterator to reduce the disk IO. How awesome is that?

libimagtag

We have exec functionality. What does this mean? It means that you can pass the commandline objects returned by the clap functions for processing the passed commandline flags and an entry and the exec functionality of the libimagtag does what the CLI tells it to do: If the user specified that there should be the tags “foo” and “bar” added but “baz” should be removed, this function does this for you.

I really want all libraries to be like this, so a developer can compose new modules by simply reusing existing functionality like this.

The CLI gets rather verbose with this, but as Rust wants it: explicit is better than implicit, and we do it just like that.

And what about modules?

We got some interesting module code merged as well. I wrote about the libraries first because the libraries are the things where the big changes happen at this moment, later the module section will come first.

So what have we done?

• imag-counter got an interactive mode
• imag-counter got listing functionality
• imag-link was initialized
• imag-store got ArgGroup
• imag-tag was initialized
• imag-view was initialized

In detail:

imag-counter

Interactive mode. That's a really nice thing in my opinion: The counter binary allows you to call interactive mode and provide a specification where you specify which key is bound to which counter. Then, you can type abc (optionally prefixed by +) to increase the counters bound to a, b and c. To decrease them you can prefix with -.

imag-link

The imag-link binary is a helper to link two entries. Not more and not less. External linking is not supported yet, though.

imag-store

This was nothing special. I merged a enhancement to make use of the ArgGroup feature clap offeres. It prevents the user from specifying two (or more) arguments which are mutually exclusive.

imag-tag

The imag-tag binary is a frontend for the libimagtag. It allows the user to add and remove tags from entries.

imag-view

Finally, the imag-view binary helps you viewing entries.

Bugs everywhere? Bugs nowhere!

So far there haven't been that many bugs! We only saw some minor bugs which resulted in rather simple errors. Nothing to serious, really.

What's in the pipeline

There are some minor things which have to be done:

• The imag-tag binary really should use the new exec functionality of the libimagtag to add/remove tags from entries
• A libimagrt functionality to call the $EDITOR must be merged. A friend of mine implemented this, thank you very much, Kai!
• The imag-view binary is implemented without library. A libimagview shall be created and used to reimplement imag-view. Actually a friend of mine volunteered in doing this. Awesome!

And of course there have to be more modules. By now, imag-counter is the only real-world (and not core- or utility-module) which is available. I'm working on imag-notes right now and I have some promising-looking code floating around for a imag-task binary implemented using “taskwarrior”. I started to write a library for writing taskwarrior hooks in Rust which I want to use to implement imag-task. The taskwarrior hook library is not ready and not published, yet.

After these two, I will start working on imag-diary, maybe imag-bookmark and either imag-contact or imag-calendar depending on what crates are available to do icalendar or Maildir stuff. If neither is available, I will do imag-password using pass (after writing another crate to interface with it, of course).

So, a lot of awesome things are planned. Stay tuned!

tags: #linux #open source #programming #rust #software #tools #imag

This is the second article on what is coming up in imag.

In the last 14 days was not that much noise in the repository, though some interesting things happened anyways.

What happened

In the following sections I will describe the PRs that have been merged in the last 14 days.

In general: There were a lot of documentation improvements and also two rather critical bugs have been fixed.

libimagtag/init

The initial codebase for libimagtag was merged. Finally, we have tagging support in form of a library! libimagtag includes a trait Tagable which is implemented on EntryHeader and Entry, so a user of the library can simply call some_entry.add_tag(some_tag) to add a tag.

The user interface for this library is rather simple, though we have now moved some functionality into a library which grants us compatibility over several modules, as entries from all modules now get tagged in the very same way.

This will also come in handy when filtering for tags, for example.

libimagstore/clean-warnings

Some warnings have been removed from the store compiling step, which results in better code quality. These warnings were nothing serious (as the development of the whole suite is not in a usable state, yet), but they will of course be removed (and turned into errors) before the first release of the suite.

doc/alphabetical-ordering

The documentation on the modules is now in alphabetical order. The non-alphabetical order becomes an issue as the number of modules slowly increases, so I switched it to alphabetical order.

doc/linking

The chapter on how linking between modules should work, was added. Finally we have something we can refer to when implementing the linking library, yay!

Of course, the implementation of the linking library was started as well and is in good progress so far.

libimagrt/config-in-toml

This was a major change. The configuration file was in some ini-like format before and now we simply use TOML. We use TOML already internally for our header format and it turned out to be great (and toml-rs, the rust library for toml is awesome as hell), so why not also using it for our configuration file format?

This removed a dependency for the config crate and we already have the knowledge how to toml-rs crate works, so there is no overhead in learning another API!

doc/add-module-files

This added simply some files for documentation of modules. Nothing too serious here.

doc/libimagutil

This added documentation for the libimagutil, which already has some really nice and useful functionality and will get some more as we make progress, so I started to add the appropriate documentation notes in the document.

libimagutil/ismatch-util

This was an awesome piece of rust macro!

It happens that we often do things like

if match foo { Value::String(_) => true, _ => false } {
    // something
}

in our rust code, when using toml-rs for example. One rather complex thing which came up just today (and is therefor not yet included, but I'll write about the problem here) was

if match foo { Some::Type::A(_) => true, _ => false } ||
    match foo { Some::Type::B(_) => true, _ => false } {
    // something
}

The macro which was introduced by this merge simplifies this noisy mess. The following is equivalent to the upper:

if ismatch!(foo, Some::Type::A(_)) || ismatch!(foo, Some::Type::B(_)) {
    // something
}

It is way more readable, isn't it?

libimagstore/update-glob

The glob crate dependency was updated as the 0.2.10 release had some bugs (mainly the PatternError which did not implement std::error::Error) which got solved in 0.2.11.

Thanks to the author for fast response times on issues and questions!

libimagstore/fix-addimplicitdirectory_creating

This was a bugfix, as the store did not yet create directories inside the store.

I started to write a module for imag and tested it by creating a foo/bar and it failed silently. I checked why and it was because the store library did not create the foo directory inside the store. Fixed that and it worked, so I extracted the patch from my module feature branch and submitted it as own PR.

libimagstore/revert-idinstore-check

This was also a hard one. There was a checking function in the store which tried to validate that a store id is inside the store (on the filesystem).

This function was buggy. It was not only this function, but also the algorithm which created the store ids. It generated the store ids (which are actually just PathBufs) as absolute paths, so foo would become /foo. This was a bug.

The store ids are no longer absolute but relative. The store now adds its own path as prefix for each ID and everything just works now.

What's coming up

Lets have a look what might be merged in the next 14 days.

First of all, there is, of course, the hook system. It is completed, though it is neiter reviewed nor merged, yet. I guess it might get some more tweaks but it will get merged with rather minor fixes instead of big changes. I'm really looking forward to get this into imag!

The second thing is: There are a lot of pending PRs which are for libraries. I want to introduce the linking library, a “notes” library which will be the foundation for the “notes” module, tagging filters in libimagentryfilter and libimagentrymarkup which is a utility to translate entry markup content into HTML (for viewing with a browser). All these libraries (besides libimagentrymarkup) are ready for merging as far as I can see, though there was no review, yet.

I also want to add a store feature so one can move an entry (Store::move() and Store::save_as() will be introduced for this). The interface is ready and submitted as PR, though there is no implementation yet.

There are some modules pending as well: A imag view utility for viewing content (mainly a frontend to libimagentrymarkup, though it will also make use of things from libimagentryfilter and possibly others as well), a imag tag utility to modify tags of arbitrary entries and imag link for linking arbitrary entries. There's also a pending PR for imag counter which will add a simple counter module to the imag suite.

All these modules are waiting for the libraries described above to be merged.

imag counter

The imag counter module will be a rather simple module. It supports creating, deleting and listing counters and of course incrementing, decrementing, setting and re-setting them.

This is planned for the first release.

After that, I want to enhance the functionality of the library and provide several counting mechanisms (and composability so other modules can use the library to use the imag counter functionality in arbitrary ways). I want to be able to count and get a graph which shows on which dates (or times) I did the most incrementing-steps, so the counter needs to save the dates it was incremented.

All these things are rather trivial to implement, though I want to make the interface as generic as possible (and provide some simple things like described above), which makes this task rather non-trivial. That's why this is planned for later and not implemented in the first step.

github pages

Another problem I'm working on is github pages. I want to upload the imag crate documentation to github pages, though it is not that simple as the repository does not contain only one crate but several.

I also want to include the code-coverage statistics later on, though this would be another step and would be done in a seperate PR later on.

Readme update

The README.md file of the repository will get an update in the next few days, so it will list a lot more information on what the repository contains and what to expect from the project.

Random thoughts

Lets see... We have some awesome PRs merged in the last 14 days, there are more awesome PRs in the pipeline and about to be merged... so awesomeness everywhere, isn't it?

Well, yes. I really want to get the libraries merged into the master branch. After that and after the hook system is merged, I can implement modules for imag and I'm also able to implement hooks for imag. And I can develop them in parallel, as they do not depend on each other. This is really important to me, as I do not get blocked from my own PRs!

Let's hope we get this stuff in! Maybe this weekend, as I have a hacking session with some of my friends planned for tomorrow, so lets see... maybe I can report some major progress in 14 days!

tags: #linux #open source #programming #rust #software #tools #imag

This is the first part of a series of Blog posts I want to release every 14 days or so on what is currently brewing in the imag (Pull-Request-)Queue.

I hope I can release one article for this series every 14 days – read: I hope there is enough progress so I can write something on it every 14 days.

Lately there is not that much progress, which mainly is caused by me awaiting review by others. I don't know why, but it is not happening and pinging them does not help that much.

Lets see what is floating around in the Pull-request-Queue of the repository.

#130 – “Doc mod”

This PR is already over 1 month old. It is meant to document features the imag modules might have. Something like “The calendar should be able to {add, remove, list} events” and so on.

At this moment, there are a whole bunch of sub-tasks for this, one for each module I want to introduce to imag. For the lazy, here is a list with short explanations attached:

• bookmarks
• borrow-tracker (if you give something to someone, this tracker can be used to remind you after N days that you should ask to get your stuff back. Should use the calendar and the contacts modules)
• calendar (via external tools, E.G.: vdirsyncer)
• contacts (via external tools, E.G.: vdirsyncer)
• counter (simply for counting things)
• cue cards (If you want to learn things and you like using cue cards, this module might help you. Attaching wiki entries or external references will be made easy via imag, though main focus is on cue-card organisation)
• dairy (This is rather simple, right? If the hook system works, this will be encrypted by default)
• images (via extenal tools, E.G.: git-annex)
• ledger (via external tools, E.G.: ledger-cli)
• mails (via extenal tools, E.G.: mutt, offlineimap, msmtp)
• movies (via extenal tools, E.G.: git-annex)
• music (via extenal tools, E.G.: git-annex)
• news (via external tools, E.G.: newsbeuter)
• notes (simple note-taking tool. Will be used by a lot of other modules to attach notes to your stuff)
• password (via external tools, E.G.: password-store)
• personal library (To track references and PDFs and handle bibtex (for example))
• todo (via external tools, E.G.: taskwarrior)
• weather (for saving the weather in your store, forecasts and such)
• wiki

The above are all high-level modules. Some more modules will be low-level modules, which are there to access the store in a module-generic way and which could be used by the user to modify single or multiple store entries, though the high-level modules should do this on their own. These modules are “just” interfaces to the respective libraries.

* link helper (CLI for the linking library) * tag helper (CLI for the tagging library) * git helper (not sure on this one, though) * content-to-website (CLI for the entrymarkup library, markup converter for viewing things in the browser) * store-integrity check (find dead links, etc. Not sure on this one either)

I guess this issue will be there for a long time.

#167 / #171 – Linking

167 is a PR for the documentation which defines how linking should work. 171 is the implementation of the things described in 167.

The PR (171) is ready and waits for review. The linking library tries to ensure that the entries are two-way linked and supports both internal and external linking.

All modules should use this library for linking later on, to ensure the linking works.

#174 – Tagging

174 is the PR for the tagging library. It implements a helper for adding, removing and setting tags in an entry. This was added because tagging is rather essential for a lot of modules (bookmarks, counter, cue cards, images, ledger, mails, movies, music, news, notes, personal library, todo, wiki). Some of these modules might be implemented using external tools (“todo” with taskwarrior for example). These modules might re-implement parts of the tagging mechanism of the tool to support imag. I'm not yet sure how to handle these external tools and discussion is clearly needed, of course.

The PR also includes a helper for building a commandline argument or commandline subcommand for specifying tagging actions. These can be used to build up similar commandline interfaces and parse values of them via the tagging library, so we have a clean UI and a single point of failure for tag handling.

#191 – Markup processing

Here I implement a high-level library which helps converting entry-content via different markup-processors into HTML pages.

This library shall be used to show content from the store to the user in a convenient way by translating content which was written in a markup language into HTML. The user of the library does not know what markup processor is used and she shouldn't care. The library gets an entry and spits out a chunk of HTML which can be shown to the user than.

#192 – “note” module

This is the first “high level module”. The PR implements the “note” module, though it is currently on hold, as I have to wait for three other PRs to be merged (171, 174, 191 – linking library, tagging library and markup library).

#193 – “counter” module

This is another “high level module” – The “counter” module. It also waits on three more PRs to be merged (171, 174, 191).

#200 – Hooks for the store

Well this is the most complex PR right now. It implements the Hook system for the store. The hook system offers different types of hooks, which are realized via traits, so everyone is able to implement hooks for the store.

These types of hooks are supported:

• post-create hooks
• post-delete hooks
• post-read hooks
• post-retrieve hooks
• post-update hooks
• pre-create hooks
• pre-delete hooks
• pre-read hooks
• pre-retrieve hooks
• pre-update hooks

Parallel execution of Hooks, although there might be RW-Hooks

The challenge here was to implement the traits for the hook types in a generic way, so each hook is able to either access the data it gets in a read-only wait or aquire write access on the data. This is currenly implemented via an “Accessor” type, which is an enumeration over either a read-only accessor or a read-write accessor (this is not integrated in the PR at the time of writing, as this is highly experimental). With this system in place, the hook system of the store is able to decide whether hooks should be ran in parallel or not. For example, if there are only Read-Access-Hooks in the list of hooks which shall be executed for a, say, create() call, the hook system is able to execute these hooks in parallel.

Not all types of hooks provide this feature, as some of the hooks are not allowed to alter data. For example a pre-read hook only gets the StoreId which shall be read, there is no point in altering the ID on read(). A post-read hook might alter an entry and therefor has to be able to provide an “Accessor” object, which then can be called by the store RO or RW.

Sorting hooks?

Another thing in my queue for this PR is the sorting mechanism for the hooks. One hook might be executed before another or after another. Hooks do not know of eachother – so you cannot compare them. So I had to “invent” a system how each hook can tell the store when it wants to be executed. There are five possible values:

• First
• earlier
• doesn't matter
• later
• Last

As far as I can see, this is not yet optimal and I have to come up with something better. That's the point why this stuff is not yet included in the PR 200 – it is not yet perfect IMO.

What the hook system may be used for

The hook system is meant to move functionality of the store out of the store library. We argued on having the store version controlled via git. One of us doesn't want that per default, as “It would pull in git as dependency” (I wouldn't mind). Having a hook system would enable us to provide version control via git or whatever other tool the user uses, without integrating the functionality in the store.

Encryption of store entries via gnupg would be easy, as the entry gets decrypted on reading into memory and encrypted on writing to disk. The store and the modules wouldn't even notice.

Consistency checking, integrity checking for linking, tagging or whatever else would be rather easy. All these things could be implemented without altering the core of the store library.

Some of these functionality would still be implemented in the core distribution of imag, though. I would really like to add encryption and version control support as libraries to the distribution and register the hooks for them in the store by default. This does not alter the store library, but the runtime library (which initializes the internal store object). Version control would be enabled by default. I'm not sure whether encryption should be enabled by default, though I doubt anyone would argue against encryption, as all the data which is in the store is rather personal (thus personal information management suite).

More PRs

There are some more PRs floating around in the repository right now. None of them is that large, just little enhancements and nice-to-have things, like colored output for the log, error tracing in color, a ismatch!() macro for less noise in code blocks like this:

if !foo.iter().all(|f| match f { &SomeTypeWithValues(_) => true, _ => false }) {
  // ...
}

// but simply:

if !foo.iter().all(|f| is_match(f, &SomeTypeWithValues(_))) {
  // ...
}

Lately merged

Some things were merged in the last five days, though I'd love to see more things merged. But as said, there were no reviews, so I couldn't merge things.

• The runtime library removed support for short versions of “—rtp” and “—config”, as the modules migth need “-r” or “-c” (#205)
• Regex are now compiled once, even if they are called in a loop - lazy_static!() helps with this (#206)
• clap and regex crate dependencies were updated (#207, #214/#215)
• Some cleanup was done and unsused imports were removed in libimagentrymarkup (#218)

Random thoughts

I'm currently thinking about the problem on how to integrate external tools without breaking user experence.

For example I use taskwarrior and I really love it. I really want to integrate it into imag, though I still want to be able to use the taskwarrior binary from the commandline as I'm used to. But after altering things in taskwarrior, they should be available in imag as well, without some futher hassle.

As taskwarrior stores its contents in JSON files, I could simply parse them and use this data, though I don't like that idea that much because if the format changes, I have to adapt my software. Also, linking to this content is rather complex, as taskwarrior seems to store everything in one big file rather than in one-file-one-task. Linking to tasks could become a major pain here.

The same goes for ledger-cli. ledger-cli supports putting each transaction into one seperate file, though the user has to cope with this when using ledger from the commandline. At least as far as I can tell.

None of these tools do export an API. Even if they'd export one, I don't think they would allow to re-implement the storage backend, so I could build an interface to my store implementation. And of course re-implementing their functionality is really not what I want to do.

taskwarrior has a hook system since 2.4.0 which could fit into my needs. ledger has not.

tags: #linux #open source #programming #rust #software #tools #imag

“imag-store” was merged today! This is the first commandline tool for imag – the commandline personal information management suite!

Okay, well, it is just the bare interface to the store and has no special features, as it is meant for debugging purposes only. Though, it is the first binary and we can finally see some progress at the CLI front! Yay!

What's next?

I'm writing some libraries at the moment, one for tagging entries and one as convenience layer for linking entries to internal and external resources (but this isn't fully specified yet). I'm also writing some utilities for entry headers, so you can query them from the CLI with a DSL (as well as in a module, but the query is interpreted at runtime and isn't build at compiletime which is way to complicated for me at this moment). I'm also trying to get the git layer in the store working, but we have a discussion about whether it makes sense to implement this non-optional or not. I'm strongly for integrating it, as it makes the store code way simpler and of course it is a big point of the store to have version information available.

And modules?

Nothing keeps you from writing one. Really, all the things are already there and implement. One last thing you might want to wait for is the implementation of the retrieve_copy() method of the store which is not completed yet. But all the other things are there and usable (tests are there, but I really would be careful – I do not consider the store “stable” yet)!

So what are the concrete plans?

1. Get the libraries implemented
2. Test the libraries
3. Start writing modules, the first modules will be the “notes” module and the “bookmarks” module, I guess.
4. More modules
5. More modules
6. More modules

tags: #linux #open source #programming #rust #software #tools #imag

This article is about the annoyance of programming language tutorials. First, I will argue why learning new programming languages always feels a bit like a pain to me. And then I will tell you why learning Rust was different.

I know a few programming languages. I've done things in C, Ruby, Java, Python, PHP, C++, Bash, Lisp (Racket, actually), Haskell, LaTeX, Nix, Make, Lua* and JavaScript*.

*: Only a bit, but still enough to say “I've done things in this language”

When I started learning programming, I learned Java and shortly after that Ruby. Both languages have their oddities, but besides their overall concept (imperative programming and object orientated programming) they are both rather easy to learn. After that I learned C. Learning C was a new world for me back then, as I only knew object orientation. I cannot remember which resources I've been using learning C, though I remember it was a pain. I had a good friend helping me, which was awesome, though I struggled a lot with the new concept.

After learning C, I did a fair amount of PHP. Learning PHP was simple (I actually did it in one weekend and a few thousand lines), though it again was a pain. It was a pain because of the tutorials and resources. And that's the point I want to discuss here.

All other programming languages I've been learning so far were painful to learn. Not because of their concepts (see Haskell, functional programming at all), but because of the resources. Haskell is a really good example here. There is exactly one resource I enjoyed while learning Haskell, and that's “Learn you a Haskell for great good”. And why? Because it takes your hand and shows you the new world. It is good because the concept of functional programming is new for most people reading this resource. Learning JavaScript, Bash, Lua or Python sucks because you already know what an object is. You know what a function or method is and you know that the basic syntax for arithmetic is the very same as in the languages you've been using by now. But all tutorials expect you start from zero – which is fine if you're actually starting from zero, but most of the time you're not. After learning Java and Ruby I understood what OOP is (actually not after learning Java, but after learning Ruby, because Java isn't a fully object oriented language in my opinion).

So, learning Haskell was kinda amazing. Learn you a Haskell is a really great book and if you want to learn Haskell, you really should read it. But what was even a better experience was learning Rust.

So why is that? It is because Rust is a new concept and the old concept (imperative programming). Learning Rust is not like learning Python after learning Ruby. It is like Learning Haskell after learning Racket. The concept behind the language is kinda the same, though it is different in most details.

Plus, the Rust tutorials and resources are freakin' awesome. Let me repeat that. The tutorials and documentation of the tools, libraries and everything, the resources on how to do things, are awesome!

Every documentation for each library looks exactly the same, which is a huge advantage over other languages library documentations. cargo, the Rust build-tool builds the documentation locally for you, if you want that. So working offline after you fetched the dependencies or your project (also via cargo) is no hassle. Building a new project with cargo is absolutely painless, its literally just executing a command, editing a textfile and executing another command (setting up the project, writing down dependencies and meta information about the project and building the project, which fetches the dependencies).

So really, if you want to learn a new programming language, check out Rust. I haven't even talked about its language features, and I will not, as this is not the goal of this article. This article is about resources and tutorials. And my conclusion is:

All programming language tutorials suck. Except the Rust language tutorial. tags: #programming #open source #bash #c #c++ #haskell #nix #racket #rust #shell

Really, how f*ck up is this travis thing?

In my current project imag, we use travis for CI. And in one of my branches, it fails because the bash scripts I use to test a binary cannot find the actual binary.

You know what the fix is? To call tree before running the scripts!

I mean, seriousely? How much can you propably mess up before things get this weird?

Fix your shit, travis people, really!

---

Update:

I mean... really? Now, the build succeeds on one test machine but fails on the other, because the build log gets too long for travis? You are freakin' kiddin' me, aren't you?

tags: #programming #software

I did it! Just hours ago I released imag in version 0.2.0.

Read here why I did that, despite there are few things working and I do not even consider this stable or even stableish.

First of all: This is not production-ready software. Use at your own risk! This is a release for people to notice that some things work and one could start to play around with it. Do not trust imag with your personal data if you do not have backups. There are bugs. This is not perfect. This is alpha quality or pre-alpha quality software!

imag is a personal information management suite for the commandline. Its target audience are commandline- and power-users. It does not reimplement personal information management (PIM) aspects, but re-uses existing tools and standards to be an addition to an existing workflow, so one does not have to learn a new tool before beeing productive again. Some simple PIM aspects are implemented as imag modules, though. It gives the user the power to connect data from different existing tools and add meta-information to these connections, so one can do data-mining on PIM data.

How the release was done

What did I do to get to the release? Well, not that much. I checked out the master branch and created a v0.2.0 branch on it, then started to fix things up for the release. That included:

• Moving the dependencies for all crates to “0.2.0”. That means that all crates in the imag code repository do not depend on other crates from the imag repository by their path but via crates.io and the “0.2.0” version of the crate. I had to do that to be able to publish these crates. This change wasn't done in master, so we can continue to work on multiple crates at the same time and do not have to cargo publish each crate independendently before beeing able to use it in other imag crates.
• I fixed the dependency markup in imag-store (actually two times) and the version information in libimagbookmark and imag-bookmark (in two seperate commits). Why is that? Well... because nobody is perfect. I simply missed them when updating the version strings from “0.1.0” (which is generated by cargo new) to “0.2.0”.

Then I manually published libimagutil and libimagerror because they do not depend on other imag crates themselves. After that I cargo published all crates in a loop until all of them were published. In the end I cargo published bin (the imag crate itself) and .imag-documentation (the documentation crate which does nothing by itself and just depends on all other crates for building the documentation).

I removed the v0.2.0 branch and created an annotated tag for the release.

Why now

I released imag 0.2.0 today because there are only few things left in the milestone for the 0.2.0 release and all whats left is tagged with release/optional, saying that these things are optional for the release (the issues will move to the next milestone, so they might not be there anymore when you click on this link).

The commit I based the release branch on was more-or-less random. It does not mark any special point in time or something, I just started to branch off of it.

Why releasing if not ready?

Why did I do the release if the software is not ready and nowhere from stable or even stableish?

That answer is pretty easy: To have a point to refer to and to get into a cycle (a release cycle). I'm pretty confident that this version is a point where interested people (read: interested developers and possible contributors) could start playing around with the software. Clearly, bugs are there and things will break. But, as imag is a rather complex piece of software and a large project, I wanted to have a point where people can be notified “Hey, this is working(ish) and you could try it out” so they can jump on the train.

And that's really what I want. I do not want to develop this monstrous project alone. I want the collaborate on this and hear what people want and need. This project solves my personal problems with PIM, but I'm sure that others will benefit from it as well and therefor community efforts are deeply encouraged by me. Or at least I hope that I do communicate this clear enough.

What's in there

Do not use imag if you do not have backups. See above for more detailed warning note.

What we have by now:

• imag-bookmark – A tool to keep and organize your bookmarks
• imag-counter – A simple counter tool. Create, delete, increment and decrement counters
• imag-diary – A diary
• imag-link – Link imag entries with other imag entries
• imag-notes – Create notes
• imag-ref – Reference files on your Filesystem with imag
• imag-store – Core plumbing tool. Do not use at all if you do not know what this is for.
• imag-tag – Tag imag entries
• imag-todo – Todo management with imag, using taskwarrior as backend. This is not a replacement for taskwarrior, but can be used to create entries in the imag store for each taskwarrior task and then these entries can be linked with imag-link.
• imag-view – View entries from the imag store
• imag – The imag binary itself. The purpose of this is to be able to call imag view instead of imag-view.

Clearly this is not everything. We have a huge number of libraries which work under the hood of these executables, but users might not care about this.

What's coming

Well, what's coming for the next release? The 0.3.0 milestone already exists and there are a lot of things already registered in it.

When will I be there? Well, my initial goal was to release 0.2.0 before 01-01-2017, so I clearly managed to get it out in time. I also announced that I want to do 3-month release cycles, so the next release would be due by March 31, 2017. I will keep that goal as I'm really not sure how fast I will be. Maybe we can release 0.3.0 early, but as normal in open source: The next version will be there if its ready.

I hope to land a imag-mail module in the next few weeks, so one can start tracking email with imag. I also hope to land imag-annotation to be able to annotate other entries with notes. imag mv will also be a thing that should go into 0.3.0 as well as more imag-view functionality.

And, of course, there will be a ton of enhancements to the libraries in imag.

Can I use imag libraries in my project?

No.

Well, you can, but I do not guarantee any stability or API consistency or something. But if you want, you clearly can use one of the imag libraries (they are on crates.io, so why not). Just keep in mind that interfaces will change and things will break and bugs will happen. We are 0.2.0 here, not 1.x.y. Keep that in mind.

For the curious, non-developers: imag consists not of one command/binary but of many. Each of those has a library accociated with it: libimagdiary and imag-diary where the imag-diary project is only a commandline interface to libimagdiary. There are some general purpose libraries in the imag codebase – for example libimagerror which are used by all other crates for a certain thing, error handling in this case. These libraries can be used independendently of imag itself. As said above, I do not guarantee anything in regard of API stability or such, so one could use them outside of imag, but I do not encourage it.

What now?

Well, I won't change my day-to-day rhythm when developing imag. As said, the 0.2.0 release was done to get some attention on imag and to people notice that some things might work, to play around and to report issues. 0.3.0 will be the same kind of release, but with more features. I don't know when I will be confident and say “Yep, one could now start to actually use this”, but I guess that it won't be 0.5.0 or 0.6.0 (which are not even planned by now). Maybe it will be 0.10.0 – who knows.

It also really depends on whether there will be (long term) contributors or not. If some people start working on imag more frequently and I can get a small community build up around the project, imag might be in a usable state “pretty soon” (read: within 2017 or 2018).

If not, ... I don't know. It's really that kind of project that runs forever. Does that bother me? A bit, ... but working on imag is fun and a pretty nice hobby... so I don't care that much.

That beeing said... feel free to play around by installing the imag crates (and having backups) and maybe you will be in the git log of the next release?

tags: #linux #open source #programming #rust #software #tools #imag

Almost all issues for the 0.2.0 release of imag are done.

Here are some notes how I want to do releases before the 1.0.0 version of imag, which, of course, is really not there yet. But I had to think about a decent release strategy for the 0.x.y releases, so here the notes.

imag is a personal information management suite for the commandline. Its target audience are commandline- and power-users. It does not reimplement personal information management (PIM) aspects, but re-uses existing tools and standards to be an addition to an existing workflow, so one does not have to learn a new tool before beeing productive again. Some simple PIM aspects are implemented as imag modules, though. It gives the user the power to connect data from existing tools and add meta-information to these connections, so one can do data-mining on PIM data.

First, all the things I note here apply to releases before 1.x.y, so basically all 0.x.y releases.

I want to do 0.x.0 releases with a bunch of new features every 3 months or something like this. This time range is not fixed, and releases will happen if they are ready, but 3 month is a good first idea, in my opinion.

I won't declare a fixed set of features for every new version, but move around some features as I like, but I try to keep the number of changes decent. For example, the 0.2.0 release has 49 closed and 13 open issues/prs at the time of writing. As I sometimes do more PRs per issue and sometimes just one PR for one issue, or even file PRs without an existing issue in the first place, this number might vary, but I guess about 100 issues/PRs per release should be a maximum.

What I want to do, and what I already do: I mark issues as

• optional if they are optional for the release they are attached to
• soft-required if I would like to get them into the release but wouldn't mind moving them to the next release
• hard-required if they have to be in the release.

I also set a release date. hard-required issues will get my attention first, I try to implement them as soon as possible and get them ready. After that I focus on soft-required things and after that on optional things.

hard-required issues might postpone a release date. soft-required and optional issues do not. But if an issue has to move to the next release, because the release-date is near and the issues are not solved by then, they will automatically advance.

That means, if an issue is optional in 0.2.0, it will get soft-required if it has to move to 0.3.0 and hard-required if it has to move to 0.4.0.

This way, one can easily calculate when a feature will definitively be implemented in the imag codebase.

---

Another thing I try to do: Plan ahead a bit. At the time of writing, the milestones for 0.2.0, 0.3.0 and 0.4.0 exist. I won't create a 0.5.0 milestone before 0.2.0 is done, so I try to plan ahead about two releases, which equals about six months.

---

I will do y-releases (as in 0.x.y) if I encounter a really serious bug. But as we do not guarantee anything right now (see below), I doubt that will ever happen.

---

All the things I wrote above are ideas how I want to do it. I will see how things work out and we'll see whether this is doable for me or not. As I already said somewhere, I'm starting my masters degree this week and I don't know how the workload will be. I really hope I can get things implemented as planned, but I might have to move issues to future milestones if I notice that things are not doable for me in the planned time.

Also, as we are in the 0.x.y release phase, I won't give any guarantees on stability and interfaces an so on.

tags: #linux #open source #programming #rust #software #tools #imag

I recently started to learn a new programming language: Rust. And I really have to say: This is the kind of language I always wanted. Some parts are not as nice as I'd like them, though most of the language, the tooling, the compiler and everything else is just so damn cool.

But first, for the new readers of my blog:

Where I came from

I learned Java as first programming language when I entered 11th grade. I learned the stuff really fast and I started to learn Ruby right after. This was about the time of Ruby 1.9.2 and 1.9.3 – and it really was an awesome time.

Later I learned C, all the way down from “Oh my gosh, why am I getting this segfault – oh, I didn't allocate properly” to linux kernel patches. I really have to say it was fun and I enjoyed learning it, though I really don't want to implement a larger project in C. C is for systems programming deep down directly at the hardware.

On my way from 11th grade until today, I learned a fair amount of languages, some of them just really briefly, others more closely: Ruby, PHP, Bash, Java, Haskell, C, C++, Scheme/Racket, Nix, Python, Lua, SQL, HTML/CSS, SASS, HAML, Latex, JavaScript, vimscript and, of course, lately Rust.

Where I don't want to go to

I have three languages I want to write in future. Maybe four. Let me elaborate:

Haskell

Nice language, though for academic use only IMHO. I really don't want to write Haskell for production systems, if possible. I still can understand why people do this and even when people start new projects with Haskell as programming language, though I wouldn't do that myself.

PHP

Do I really have to explain this? I mean, PHP was great in the '90s and maybe early 2000s. But today, there are much better alternatives available.

HTML/CSS

Yes, you kind of have to write these languages in web development, but I'd opt for SASS and Haml. They are both nice preprocessor languages which are much simpler to use.

JavaScript

I didn't learn much about JavaScript by now, though I don't think it is a language I would want to use except for website animations.

Scheme/Racket

Nice languages! I really enjoy reading them, though I don't see any value in them for large production systems. Their tooling just looks to bad (from the outside) and while I'd love to see better tooling here, I don't really care about these languages.

Java

Nope.

Python

I see a lot of value in Python for small scripts, but not for large production software like web frameworks or similar. Yes, this was a hint in direction of Django!

Lua

Nice language for embedded stuff and so on, though I don't really care too much about Lua either.

SQL

The standard when it comes to database querying. Though I really don't want to write these things by hand and I really would love to have ORMs widely available (and yes, every major language has an ORM, I know that and I like it).

Latex

Can we please just have pandoc succeed and write our stuff in Markdown? I mean, yes, Latex kindof works, but when it doesn't it just sucks as hard as hell!

vimscript

Another thing I don't want to have to touch.

C++

There are better alternatives available... this is why I'm writing this post!

Where I want to go to

Yes, there are languages I like. It's really just three/four, but they clearly exist: Ruby, C, (Bash) and Rust. Each of them with another purpose. Bash is in parentheses here because I kindof have to use Bash, let me explain this first:

(Bash)

Bash is syntactically just disturbing. I mean, its syntax just sucks, right? But it is the default on all Unix systems and that won't change, so for scripts, I just adapt to this default and keep my things in Bash.

I know there are alternatives like fish, zsh and whatever. I cannot use them because of two reasons: If I change my shell, I want to change it everywhere, means I would need to convert all my scripts. This is a huge amount of work. And, I'd still have to write bash when contributing to nixpkgs, and not contributing to nixpkgs is just no option for me.

So, for the sake of uniformity, I just keep everything in bash.

Ruby

Ruby is awesome. I love its syntax, I love its tooling (though I don't have to mess that much with it).

I just don't want to implement complex things in it. I started to learn rails and I really like rails, despite it is a monster from my perspective. There are other web frameworks for Ruby available which are much less complicated. I speak partly of Sinatra, but also of Lotus which seems quite nice to me (again, from the outside).

Of course there are others.

C

C is,... well C. It is awesome for small bit-fiddeling work and OS kernels. It is great for microcontrollers. It is just as simple as possible a high-level-assembler language can be.

I really love writing C, though I also think one has to avoid it whenever possible.

Rust

Now I finally hit the chapter this whole blog post is about.

Rust. Is. Awesome.

I mean, really. It comes with the tooling of Ruby, cargo the package manager/build tool/test runner/everything just works out of the box and as good as one can imagine. Working with dependencies just works (if you get your types right). The language itself is really nice. I mean, I worked with for about one month by now and I really enjoy every problem I encounter in manner of a learning experience. I learn with every problem and I also learn to avoid problems really quickly.

Rust gives me power to my hands I couldn't even imagine when I was in 11th grade. I can write high level code like in Ruby, but I can also go into detail and fiddle around with bits and everything, without having to worry about References and pointers, data races, bla bla bla (take that, C++)! If the compiler tells me that I've done everything right and it compiles my code, I can be sure that the program won't eat my head while running. It might not do what I want, but it does what is written down without crashing in random places!

So, with Rust you get:

• If it compiles, it works (Haskell)
• It is fast ©
• You can build complex applications with type safety (C++ (though “type safety” is a stretchable term in C++))
• You can write high-level code (Ruby)
• Easy syntax (Ruby)

Summary

So, to summarize: Bash for system scripting, Ruby for more complex scripting tasks, C for embedded/system programming, Rust for everything else from systems programming to building space rockets.

If you haven't learned Rust yet, go and give it a try! You will be amazed!

tags: #programming #open source #rust