musicmatzes blog

Rust

This is the 9th iteration on what's coming up in imag, the personal information management suite for the commandline.

First of all, I want to thank and welcome kbknapp who contributed a Makefile which enables to build the imag tools with just one command: make. Thanks a lot for that!

What happened

I wasn't able to do that much in the last 14 days, as my bachelors thesis wants some attention as well. But I was able to merge some nice PRs anyways, so here we go.

New issues!

In the last 14 days I tried to open as many issues as possible, tagging them and therefor making them accessible by possible new contributors (as some of them are tagged with the complexity/easy tag).

Diary!

Finally, the initial implementation of the diary module was merged! Yay! It already has some features I really like, but isn't feature-complete yet and there might be bugs. But the initial codebase is merged and I'm really happy about that.

One can use imag now as a personal diary. I did not integrate some diary tool into imag by now, because I thought implementing it myself would be kind of fun. Of course, I would like to add other tools as backend but I don't know of any (and I do not use any) I would find sufficient for integrating into imag.

More libraries and library-functionality!

Two awesome PRs were merged which added new functionality.

The first one is the “move” functionality for the store. It is not yet used by any other crate, but gives the possibility to save a copy of a store entry or move a store entry to a new location within the store. This functionality is not yet tested, but as we are pre-release anyways, this is not an issue in my opinion.

The other functionality is the libimagentrymarkdown which adds markdown-parsing functionality. There is just the initial codebase for this crate added by now, but the library will make it possible to parse the contents of a store entry to a web page and view it in the browser, for example.

The other merges

I merged some rather critical bug fixes, for example one were an entry wasn't edited correctly when calling edit_content() (from libimagrt) on it. A code-cleanup PR for libimagentrylink and some other fixes were merged as well.

The future

So lets see what's in the pipeline at the moment.

I'm working on a new crate called libimagref which gives access to primitives and functionality to reference files which do not live inside the store. For example to reference some music files, the libimagref can be used to generate a reference in the store which tracks the file(path), optionally having the possibility to check the metadata of a file by generating a hashsum over its contents and/or permission settings.

Some tests for the tagging binary (imag-tag) are work-in-progress.

For the next 14 days I hope to finish the libimagref implementation as well as getting some progress in the libimagentryfilter/lang PR, which aims for a DSL implementation in the entry filter crate, giving the user access to a filter language to filter out entries on the commandline like so (diary example here):

imag diary list --filter 'e.size() > 5.kilobytes() && e.lines().first().empty()'

Of course, the upper example is not how it will look like, as I'm still not sure what DSL implementation I will use and how I will model the access to the entry data, but the idea will be the same.

Another thing I really want to test out is quickcheck. I guess this could be really useful to test imag library crates properly and I hope to get something up in the next 14 days in this regard.

tags: #linux #open source #programming #rust #software #tools #imag

Another 14 days are over. Here's my update what's coming up in imag, the personal information management suite for the commandline.

The past

We had 40 closed issues and pull-requests in the last 14 days, which is really awesome.

I was able to merge 29 pull requests in the last 14 days, which is quite a lot, so I won't elaborate on all of them (some of them are just small fixes), but let me do a short iteration on the highlights:

  • We implemented Store::get() which returns a None if the entry does not exist (Store::retrieve() implictely creates the entry).
  • The Store::get() functionality was also implemented in the imag store, imag notes and imag tag commands, causing them to fail if you try to list or view non-existent store content.
  • We have a new Hook which gets executed by the store after the store is loaded (so it is called only once, but hooks might do some fancy stuff before the user commands are executed).
  • We have a new crate for writing Date and Time interfaces, which helps parsing user-specified date and time. This crate is not tested yet, though some tests exists within the crate and they succeed.
  • Another crate was introduced for querying the user to provide a store id if he hasn't already, providing a fuzzy-search interface based on the interactor crate. This crate is not tested yet.
  • The error crate libimagerror provides a way to define custom members and custom code for the error types now, beeing more flexible now.
  • libimagerror also implements Into for the error kinds now, making the code using it even less complex.
  • All crates were rewritten to use the infrastructure provided by libimagerror
  • The binary at bin/imag was rewritten from Bash to Rust.
  • Updates for some dependencies were done
  • Bug fixes, including but not limited to
    • A bug in the config-finding algorithm in libimagrt
    • Bugs in the Hook execution in libimagstore
    • Duplication of data in libimagstore (FileLockEntry and Entry had the StoreId stored, despite the one dereferences to the other)

I also had (finally) some progress with my diary implementation. I guess I may finish it within the next 14 days. I'm serious this time, as the current code works and almost all the things are implemented at the time of writing.

I also have some (little) progress with the bookmark implementation, though I'm not sure whether I will implement this (rather) simple tool by myself or re-use existing solutions. On the one hand the problems are rather simple to solve with the existing imag infradstructure, on the other hand, I don't want to double work. I guess I will end up implementing it myself anyways, as I consider the (external-)link code part of the infrastructure and I don't want to depend on an external tool which might be or get unmaintained at some point in future.

I also have a new crate in my pipeline, which helps with Markdown parsing. So for the first step, we will provide Markdown only. But I will wrap this crate libimagentrymarkdown into a more generic one (libimagentrymarkup) to be able to write more backends for it (thinking of textile, asciidoc and so on, maybe by using a pandoc interface) and support other markup languages later on without hassle.

All in all, I really made some progress with the project in the last 14 days. I wrote more than 80 commits in the four days of the GPN last weekend, which was a huge amount of the work in the last two weeks, of course. I hope I can continue my streak.

The future

Well, the future. I don't really think I can keep up with my latest streak, though I really hope getting the diary implementation done. Bookmarks would be nice as well.

One thing I'm thinking about almost constantly in the back of my head is the DSL implementation for libimagentryfilter. I really want to dig in a little deeper into this. Maybe I can get something working within the next 14 days and if it proves useful I might extend the DSL so I can script the Store with it. Of course, this won't be a replacement for Rust as implementation language, but might prove useful for smaller scripts and the like? I'm not sure.

I also hope to find some contributors for the project. If you are looking for rust projects to contribute to, feel free to ask me questions or simply try to implement one of the complexity/easy tagged issues on github. Of course you can also ask questions there, I'm happy to answer them!

tags: #linux #open source #programming #rust #software #tools #imag

Welcome to the 7th post about the last 14 days in imag, the personal information management suite for the commandline written in rust.

We had 22 merges in the last 14 days, 24 in the last 15 days, since the last article of the series.

Lets see what happened.

Merged pull requests

Next goes a short explanation on each of the merged pull requests from the last 14 days.

#339

The StoreId Type was rewritten and isn't a pub type StoreId = PathBuf anymore. This change should be fully backwards compatible, so nothing will change, though we have the possibility to implement custom traits on the type from now on, which can make things easy in some situations.

#385

This merge fixed a typo in the Readme file. Thank you, Ron, for contributing to imag!

#349

The libimagstore got a Walk Type which can be used to walk whole store directories. It is based on the walkdir crate and also iterates over directories while recursing into them as well. So one can use this functionality to explore the collections within a module storage.

#387

This merge removed some old code which did some configuration file checking but was wrong.

#389

This merge added an example configuration file to the repository.

#388

This was also a bugfix which fixed some missing error handling in the configuration file parsing code.

#390

This was a mass-update of the log crate dependency for all crates in the repository.

#393

With this merge we updated the url dependency of libimagentrylink

#394

We made a helper function which could be used to filter links to external link files public with this merge.

#395

This was one of the greater PRs.

With this merge, we made the StoreIdIterator generic. Before this merge, the StoreIdIterator type was a wrapper around the Paths Iterator type from the glob crate. This wasn't really good, as we couldn't use the StoreIdIterator type to iterate over a Vec<StoreId> for example. This merge makes this now possible.

#384

llogiq made some improvements to the imag code base by running clippy on it and removing the warnings clippy yielded. Thank you very much for contributing to the imag codebase!

#401

General code cleanup in libimagutil.

#404

With these patches merged we now have the possibility to specify multiple tags when using imag-tag to add or remove tags.

#402

General code cleanup in libimagrt.

#403

General code cleanup in libimagstore.

#412

We have now colored log output with libimagrt! Awesome!

#411

Some improvements to the CONTRIBUTING.md file.

#413

Error tracing prints the errors now in red (blinking). I really like that!

#410

A new library got into the imag codebase: libimagerror contains code to handle imag error types.

#415

This is a follow-up patch for #410 to move the trace_error() functions from libimagutil to libimagerror.

#417

This merge added a function in libimagrt to generate the Runtime setup or exiting on failure. Just convenience and less boilerplate code in the binaries. Yay!

#418

Here we added some helper functions in libimagentrytag for parsing the commandline options.

#422

matthiasbeyer/libimagerror/generate-module

Another follow-up PR for #410. With this merged, we generate a full error module when using the macros from libimagerror. This way, we have more flexibility in generating our error code.

#416

Aaaand another follow-up PR for #410. With this merged, we can call into_error() on all error kinds to generate the appropriate error object out of them and into_error_with_cause(...) to pass a cause. This will lead to more readable code everywhere.

Maybe another follow-up will follow which then removes the noisy FooError::new(FooErrorKind::SomeErrorKind, Some(Box::new(other))) and simply put FooErrorKind::SomeErrorKind.into_error_with_cause(Box::new(other)), which is way more readable IMHO.

What's coming up

I hope we can continue these libimagerror patches and add the ability to add custom code in the error types. We need this for #370 as we add some custom members in the HookError Type in this PR.

I'm not really sure what else will be done in the next 14 days. I am still working on the diary implementation (actually with little to no progress in the last 14 days). Because of this I'm not sure what I can get done in the next 14 days.

I really hope to get some more tests merged. One PR (tests for imag-tag) is already pending. I also want to get some markup-parsing library working (I have a pending PR for this which is already three months old).

If you think you could contribute some code to imag, feel free to do so, there are a lot of open issues which are tagged easy!

tags: #linux #open source #programming #rust #software #tools #imag

Two new contributors! Yay!

Another 14 days in imag, lets start iterating over the latest changes (spoiler: there are not that much).

The past

As said, there are not that much changes in imag in the last 14 days. Why is that? Because I have a whole lot to do for my university and some events I'm organizing. The event takes place this week, so after that I hope I can focus more on imag (and my bachelors thesis, hopefully).

But hey, anyways, things happened. Lets iterate.

Fixes!

We had some fixes in the last 14 days and some more are in the pipeline. We removed some result.err().unwrap() calls in the error handling part of functions and replaced it with result.unwrap_err(). Kind of cosmetic, but anyways an improvement in my opinion.

Testing

I merged a PR for imag-link testing. Still, we are testing by using bash scripts, which is what I really want to improve. I want to rewrite the tests in Rust to be more safe.

Hooks

I merged a new hook: A hook to verify links in the entries which are about to be altered or touched by a store call. You can use it as pre-update hook or as pre-retrieve hook.

The hook verifies that the entry which is about to be touched links to entries which actually exist on the filesystem.

Warnings

I removed almost all warnings which occured when compiling the imag codebase. This is awesome, isn't it? I enabled some lints and compilation will now fail if someone screws up. I love it!

The future

I got some PRs. Awesome!

PR 383 is a PR for implementing libimagtask and imag-task, using task-hookrs as backend to talk to taskwarrior. I will, of course, support this as much as possible with both patches and suggestions.

PR 384 is a PR from llogiq, one of the authors of clippy. It fixes some issues with the codebase overall. So awesome!

These PRs will be merged as soon as they are ready.

I'm not sure what else I can do in the next 14 days, but I want to go on implementing the diary and maybe get the filter language working, which is a rather complex task. Maybe I will implement the bookmarking module by adapting another tool or maybe by implementing it myself.

tags: #linux #open source #programming #rust #software #tools #imag

Wow, another 14 days! When I started writing this article, I thought I didn't manage to do a lot of changes in the last 14 days. git proved me wrong.

What happened

When I started writing this article, I thought “Oh man, another 14 days with little progress in imag”. I was proven wrong:

$ git log --oneline --merges --since=2016-04-07
c3618ec Merge pull request #343 from matthiasbeyer/libimagstorestdhook/flock
45f93e5 Merge pull request #366 from matthiasbeyer/update-dependency/regex
25a71ce Merge pull request #365 from matthiasbeyer/libimagstorestdhook/zero-warnings
a2288ba Merge pull request #364 from matthiasbeyer/libimagnotes/zero-warnings
791b3ad Merge pull request #363 from matthiasbeyer/libimagentrylist/zero-warnings
f0d29da Merge pull request #362 from matthiasbeyer/libimagentryfilter/zero-warnings
e040d0d Merge pull request #361 from matthiasbeyer/libimagrt/zero-warnings
3e29ec0 Merge pull request #359 from matthiasbeyer/remove-init-logging
19808cf Merge pull request #360 from matthiasbeyer/update-dep/iterools
99e1e41 Merge pull request #353 from matthiasbeyer/rename/libimaglink
a5e9bf7 Merge pull request #358 from matthiasbeyer/imag-view/zero-warnings
d3b94df Merge pull request #354 from matthiasbeyer/libimaglink/zero-warnings
3a5c965 Merge pull request #355 from matthiasbeyer/libimagutil/zero-warnings
86ae438 Merge pull request #356 from matthiasbeyer/imag-counter/zero-warnings
f64ed8e Merge pull request #357 from matthiasbeyer/imag-link/zero-warnings
ed6d912 Merge pull request #352 from matthiasbeyer/rename/libimagtag
179ae2b Merge pull request #350 from matthiasbeyer/libimagtag/zero-warnings
f003625 Merge pull request #351 from matthiasbeyer/libimagcounter/zero-warnings
ff20e79 Merge pull request #338 from matthiasbeyer/imag-link/external-linking
e4e3c05 Merge pull request #326 from matthiasbeyer/libimaglink/external-linking-rewrite
3d124df Merge pull request #347 from matthiasbeyer/libimaglink/unique-internal-links
17610ef Merge pull request #337 from matthiasbeyer/imag-link/list-implementation
20a76f8 Merge pull request #330 from matthiasbeyer/update-contributing
535483d Merge pull request #335 from matthiasbeyer/libimaglink/bugfixes
18627f1 Merge pull request #336 from matthiasbeyer/imag-link/cli-unpacking-bugfix
8d7e2d9 Merge pull request #331 from mario-kr/add-description_to_build/run_imag
9866c63 Merge pull request #311 from matthiasbeyer/libimaginteraction/init
92b471b Merge pull request #328 from matthiasbeyer/imag-link/fix-ui
cd2bc0b Merge pull request #325 from matthiasbeyer/libimaglink/internal-linking-use-storeid

Let me do a short iteration.

Zero Warnings

I opened an issue about warnings and that we should enable compiler lints to make them hard errors. Then I started to write pull requests for each crate in the repository and also got them merged. We have almost no warnings left when compiling imag, which is what I consider nice progress!

Init-logging

Some time ago we rewrote the Runtime setup code for logging, so we had the logger enabled at the earliest possible point in program execution. Now I removed some debug!() calls which were done in the main.rs of almost all crates right after the Runtime object was built which stated something like "logging enabled...". These debug!() calls were outdated, so I removed them.

Libraries renamed

Some of the libraries were renamed. For example "libimagtag" got renamed to "libimagentrytag".

This is none of a big deal and as we are pre-1.0 I can do this, I guess.

This is one of the gems here, really. I finally got my feet up to rewrite the external linking part of the linking library and got imag-link adapted so it works appropriately now. How awesome is that?

More Hooks!

I implemented a flock() hook in libimagstorestdhook and I have another open PR for a verify-link hook which uses libimagentrylink to verify that all linked entries exist.

Dependencies updated!

I updated a number of dependencies (namely itertools and regex) and I also rewrote the Cargo.toml files to depend not on the master-minor-patchlevel version of the crates but on master-minor versions, so cargo can figure out what patchlevel to use.

What's coming up

I'm about to re-write the libimagdiary because what I did in my initial PR for the library was a mess. I tried to implement the diary functionality by re-using the note functionality, which resulted in ugly spagetti code and the like, so I decided it would be better to just re-write the whole thing.

I'm about to merge a PR which introduces the concept of non-failing hook errors. So a hook can return an error which does not abort the store operation which invoked the hook itself.

I also started to implement libimagentryview on my own, as the friend of mine who said he wants to do it hasn't made anything yet, so I'm a bit pissed of on him. But Hey, it is open source, nobody can force anybody! So I will just stick with implementing it on my own (and it is also almost finished).

I also started to refactor the implementation of the StoreId type, as we might need to extend this type in external crates, so doing a pub type StoreId = PathBuf was not the best idea and I started to rewrite it as pub struct StoreId(PathBuf) – I hope I can manage to do this in a backwards compatible way, but I'm not sure.

I also started implementing a Walk type for the Store which does a walk through all the objects in the store. This is not an iterator but something that also notes folders within the store and gives the possibility to iterate through a tree rather than through a flat list of entries. Could be helpful sometimes.

We also have an open PR for DSL support with libimagentryfilter, so the user is able to filter entry iterators with a DSL. This is early-state but looks rather promising IMHO.

tags: #linux #open source #programming #rust #software #tools #imag

Another exciting two weeks! This is the fourth iteration on what's coming up in imag, my personal information management suite for the commandline.

And we had an awesome bunch of merges in the last 14 days!

Editors everywhere!

We now have a utility function in the runtime library for calling the $EDITOR, which either uses the editor specified by the commandline arguments, or the configuration, or the $EDITOR variable from the environment.

This was implemented by a friend of mine who wants to learn rust as well. Thanks a lot!

Warnings? No, Errors!

We got two patches merged (one for the store library, one for the runtime library) which turned warnings of a certain kind into hard errors and prevent compilation. I take this as “We get more stable”.

More debug output!

A whole bunch of merges were done to enhance the debug output in a lot of cases. For example, the Store object implements Debug now, which means that debugging the store is way simpler than before.

Bugfixes!

Remember me writing about how few bugs we had by now? Well... we had some but did not find them. We merged a bunch of bugfix-pull-requests in the first part of the last 14 days. Most of them were rather simple bugs (one was even a one-character-was-wrong bugfix) but nevertheless critical.

Readusall!

I added one readme file for each crate in the project, so people can see what are the libraries supposed to do. Awesome, isn't it?

New Issues!

I've added a whole bunch of issues in the last 14 days, and all of them are tagged with either complexity/easy, complexity/medium or complexity/high as far as I can remember. So if you want to have a look what's going on and maybe help with some pull requests, the ideas are already there!

Here's a short iteration of the issues I opened...

  • #282 - libimagrt config returns ConfigNotFound even if it was a Parser error. The runtime library returns a RuntimeErrorType::ConfigNotFound even when there was a parser error. This has to be fixed. Complexity: Easy.
  • #284 - imag-tag should use libimagtag::exec::exec_cli_for_entry(). The imag-tag module does not yet use the libimagtag functionality for executing the commandline specification via the library and instead reimplements this functionality. Complexity: Easy.
  • #294 - Store should automatically fallback to latest version of entry. The libimagstore::store::Store type does only load the entry specified. By using the libimagstore::storeid::StoreId type and the generator macro for generating these objects, we enforce the version of the entry to be present. What we do not do is to allow the version part to be optional (when building the entry path from a commandline specification). If we would allow this, the store would need to fall back onto the latest version of the entry which is currently available. The store does not do this at the moment, but I think this would be a nice feature, actually. Complexity: High.
  • #297 - Library for extracting links from content. There has to be a markup library which allows us to extract links from a Markup document such as Markdown, TexTile and so on. This library and libimaglink could then be used to create the header links via some store hooks. Complexity: medium / maybe high.
  • #298 - Hook: Verify that all linked entries exist. That's a rather simple one: We need to check whether all links from a header are actually existing files in the store. We want to do this as hook and we want the hook to be configurable to either deny entry changes or just print warnings.
  • #299 - Check whether we want ot use Cow in our APIs. Another rather simple one: We want to check whether it would be beneficial if we'd use Cow in our internal APIs. Mostly for String types, but maybe also for other types as well. Complexity: Easy.
  • #300 - Rewrite libimaglink – external linking. The external linking library module has to be rewritten. See the issue for more information. Complexity: Medium.
  • #301 - module: Bookmarks. We want to introduce the bookmarking module. Complexity: Easy.
  • #302 - Store Hook Aspect configuration: Setting do deny mutable hooks. We want to be able to configure hook aspects to deny mutable hooks. This is mainly a security-related task and would be a nice to have. Complexity: Medium.
  • #303 - Make store hooks be executed in parallel if possible. We want to alter the hook execution algorithm to execute hooks in parallel if possible. Complexity: High.
  • #304 - imag-view: libimagentryview has to be written. We want to extract the viewing functionality from imag-view and add some more functionality to it. This should be put into a new library libimagentryview. Actually there's someone working on this. Complexity: Easy.
  • #306 - module imag-stat. Another nice-to-have: A module for getting some stats/diagnostics about the store. Complexity: Medium.
  • #307 - libimaglink: XDG-open for external links. We want to add a feature in libimaglink to open external links via xdg-open. Complexity: Easy, can only be started after #300 was merged.
  • #308 - Logging: Print file and line on debug!(). We should alter our logger backend implementation to print file and line on each debug!() log call as well. Complexity: Easy. This Issue was closed at the time of publishing this blog article.
  • #312 - Rename: libimaglink –> libimagentrylink. We should rename this library for more consisteny. Complexity: Easy.
  • #313 - Rename: libimagtag –> libimagentrytag. We should rename this library for more consisteny. Complexity: Easy.
  • #314 - Remove warnings from libimagutil. We should remove the build-time warnings from this library. Complexity: Easy.
  • #315 - Convert TOML destructuring into toml::Value::lookup(). As the TOML crate supports Value::lookup() to lookup values at a specific path, we can use this functionality to replace a lot of boilerplate code which simply destructs TOML objects. Complexity: Medium, because a lot of code has to be checked and altered. The changes themselves might not be that complex.
  • #316 - module: weather. I added this issue for tracking of the implementation of the weather module. Yes, we plan a weather module for imag, see the linked issue what I expect such a module to do. Complexity: Easy.

This list could be seen as a cry for contributors! At least there is now a list available what has to be done and with some complexity level attached - contributors can now see where to start!

Side-Project: task-hookrs

This is a little side-project which I started. It aims to implement a crate for interacting with taskwarrior, so you can export tasks from taskwarrior and import them into your rust program. Of course I'm writing this so I can use it in imag to implement the todo module by using taskwarrior as todo manager and interface with it through taskwarrior hooks and my task-hookrs crate.

In the last 14 days I got some important points working: Deserializing and Serializing are already working and so is importing from a Read. Exporting is on its way.

Some things I didn't implement yet is support for the id and urgency fields and annotations are missing as well, but I guess these are really simple to implement as well.

I'm really looking forward to get this crate working in the next days and I'm starting to wonder whether I should put in on crates.io and how this whole process works.

What's coming up

As always, I want to close this article with a prospect. I got less stuff done than I expected in the last 14 days:

$ git diff --shortstat $(git log --since=14days --format=%H | tail -n 1)
56 files changed, 1193 insertions(+), 86 deletions(-)

I'd love to get some more progress in the next 14 days, though I cannot guarantee it, as I have to write a bachelors thesis and as I have to do some other important things as well. I really hope I can get the diary module ready and merged within the next days and maybe I can start implementing the bookmark manager then (after I solved #300). As this is not that complicated besides the #300 which has to be merged first, I might get started with the todo module as well. I really hope so.

tags: #linux #open source #programming #rust #software #tools #imag

The last 14 days were really exciting. We merged some awesome things and even more awesome things are in the pipeline, but lets focus on the past two weeks in this third update on imag, the personal information management suite for the terminal.

Hooks everywhere!

First of all, the big news: The hook system got merged.

Holy crap, this was a big one! 950 lines were added and 10 were deleted by this merge, 14 files were changed. Well, I do not want to elaborate on the architecture of the hook system now, this really belongs to the documentation. What I want to do in the next 14 days is implementing a libimagstorestdhook library, which contains several builtin hooks for the store, such as:

  • A rather simple “Debug”-Hook, which does nothing but printing debug information about the hook execution.
  • A VCS hook which uses git to make the store version-controlled
  • A crypto hook which encrypts and decrypts the store on the fly.

For the first iteration, the debug hook is enough. The other hooks are nice to have, though I would only create the stubs for them and then focus on even more important things: modules.

Where are my modules?

Well, I got some really interesting things merged in the last 14 days. Some libraries were merged which really make writing modules a joy. Here is the short list:

  • libimagentryfilter got a version filter
  • libimagentryfilter got some more rather basic filters added
  • libimagentryfilter got the ability to filter by tag
  • libimaglink initial codebase was merged
  • libimagstore got a function to retrieve all entries for one module
  • libimagtag got exec functionality

But what does this mean in detail? Let me elaborate:

libimagentryfilter

We have two new filter types in this library. First, the version filter. It enables to filter out entries by their version field from the header, so you can, for example, select only entries which are in a certain range of versions, if that matters to you when implementing a feature for a module. This filter must prove its usability, of course.

The second filter is a tag filter, so you can select entries by tags. There's no CLI functionality for this, yet. I really want to implement this as well.

Some more basic filter functionality was added. The very core of these changes is a filter which takes a predicate closure, all other basic filters (such as the header-field-is-equal-to-filter and so on) were reimplemented using this core filter type.

Yeah, we finally got this code merged! We have a library for linking entries now. I already elaborated this in the last part, so I won't elaborate on it here

libimagstore

This is also really really nice. We have a Store function to retrieve all entries for one module now. And the best thing is: It is lazy, because it returns an iterator! It uses the glob crate internally, which returns an Iterator itself and we wrap this iterator into a StoreIdIterator. With libimagentryfilter you can simple call filter() on this iterator to reduce the disk IO. How awesome is that?

libimagtag

We have exec functionality. What does this mean? It means that you can pass the commandline objects returned by the clap functions for processing the passed commandline flags and an entry and the exec functionality of the libimagtag does what the CLI tells it to do: If the user specified that there should be the tags “foo” and “bar” added but “baz” should be removed, this function does this for you.

I really want all libraries to be like this, so a developer can compose new modules by simply reusing existing functionality like this.

The CLI gets rather verbose with this, but as Rust wants it: explicit is better than implicit, and we do it just like that.

And what about modules?

We got some interesting module code merged as well. I wrote about the libraries first because the libraries are the things where the big changes happen at this moment, later the module section will come first.

So what have we done?

  • imag-counter got an interactive mode
  • imag-counter got listing functionality
  • imag-link was initialized
  • imag-store got ArgGroup
  • imag-tag was initialized
  • imag-view was initialized

In detail:

imag-counter

Interactive mode. That's a really nice thing in my opinion: The counter binary allows you to call interactive mode and provide a specification where you specify which key is bound to which counter. Then, you can type abc (optionally prefixed by +) to increase the counters bound to a, b and c. To decrease them you can prefix with -.

The imag-link binary is a helper to link two entries. Not more and not less. External linking is not supported yet, though.

imag-store

This was nothing special. I merged a enhancement to make use of the ArgGroup feature clap offeres. It prevents the user from specifying two (or more) arguments which are mutually exclusive.

imag-tag

The imag-tag binary is a frontend for the libimagtag. It allows the user to add and remove tags from entries.

imag-view

Finally, the imag-view binary helps you viewing entries.

Bugs everywhere? Bugs nowhere!

So far there haven't been that many bugs! We only saw some minor bugs which resulted in rather simple errors. Nothing to serious, really.

What's in the pipeline

There are some minor things which have to be done:

  • The imag-tag binary really should use the new exec functionality of the libimagtag to add/remove tags from entries
  • A libimagrt functionality to call the $EDITOR must be merged. A friend of mine implemented this, thank you very much, Kai!
  • The imag-view binary is implemented without library. A libimagview shall be created and used to reimplement imag-view. Actually a friend of mine volunteered in doing this. Awesome!

And of course there have to be more modules. By now, imag-counter is the only real-world (and not core- or utility-module) which is available. I'm working on imag-notes right now and I have some promising-looking code floating around for a imag-task binary implemented using “taskwarrior”. I started to write a library for writing taskwarrior hooks in Rust which I want to use to implement imag-task. The taskwarrior hook library is not ready and not published, yet.

After these two, I will start working on imag-diary, maybe imag-bookmark and either imag-contact or imag-calendar depending on what crates are available to do icalendar or Maildir stuff. If neither is available, I will do imag-password using pass (after writing another crate to interface with it, of course).

So, a lot of awesome things are planned. Stay tuned!

tags: #linux #open source #programming #rust #software #tools #imag

This is the second article on what is coming up in imag.

In the last 14 days was not that much noise in the repository, though some interesting things happened anyways.

What happened

In the following sections I will describe the PRs that have been merged in the last 14 days.

In general: There were a lot of documentation improvements and also two rather critical bugs have been fixed.

libimagtag/init

The initial codebase for libimagtag was merged. Finally, we have tagging support in form of a library! libimagtag includes a trait Tagable which is implemented on EntryHeader and Entry, so a user of the library can simply call some_entry.add_tag(some_tag) to add a tag.

The user interface for this library is rather simple, though we have now moved some functionality into a library which grants us compatibility over several modules, as entries from all modules now get tagged in the very same way.

This will also come in handy when filtering for tags, for example.

libimagstore/clean-warnings

Some warnings have been removed from the store compiling step, which results in better code quality. These warnings were nothing serious (as the development of the whole suite is not in a usable state, yet), but they will of course be removed (and turned into errors) before the first release of the suite.

doc/alphabetical-ordering

The documentation on the modules is now in alphabetical order. The non-alphabetical order becomes an issue as the number of modules slowly increases, so I switched it to alphabetical order.

doc/linking

The chapter on how linking between modules should work, was added. Finally we have something we can refer to when implementing the linking library, yay!

Of course, the implementation of the linking library was started as well and is in good progress so far.

libimagrt/config-in-toml

This was a major change. The configuration file was in some ini-like format before and now we simply use TOML. We use TOML already internally for our header format and it turned out to be great (and toml-rs, the rust library for toml is awesome as hell), so why not also using it for our configuration file format?

This removed a dependency for the config crate and we already have the knowledge how to toml-rs crate works, so there is no overhead in learning another API!

doc/add-module-files

This added simply some files for documentation of modules. Nothing too serious here.

doc/libimagutil

This added documentation for the libimagutil, which already has some really nice and useful functionality and will get some more as we make progress, so I started to add the appropriate documentation notes in the document.

libimagutil/ismatch-util

This was an awesome piece of rust macro!

It happens that we often do things like

if match foo { Value::String(_) => true, _ => false } {
    // something
}

in our rust code, when using toml-rs for example. One rather complex thing which came up just today (and is therefor not yet included, but I'll write about the problem here) was

if match foo { Some::Type::A(_) => true, _ => false } ||
    match foo { Some::Type::B(_) => true, _ => false } {
    // something
}

The macro which was introduced by this merge simplifies this noisy mess. The following is equivalent to the upper:

if ismatch!(foo, Some::Type::A(_)) || ismatch!(foo, Some::Type::B(_)) {
    // something
}

It is way more readable, isn't it?

libimagstore/update-glob

The glob crate dependency was updated as the 0.2.10 release had some bugs (mainly the PatternError which did not implement std::error::Error) which got solved in 0.2.11.

Thanks to the author for fast response times on issues and questions!

libimagstore/fix-addimplicitdirectory_creating

This was a bugfix, as the store did not yet create directories inside the store.

I started to write a module for imag and tested it by creating a foo/bar and it failed silently. I checked why and it was because the store library did not create the foo directory inside the store. Fixed that and it worked, so I extracted the patch from my module feature branch and submitted it as own PR.

libimagstore/revert-idinstore-check

This was also a hard one. There was a checking function in the store which tried to validate that a store id is inside the store (on the filesystem).

This function was buggy. It was not only this function, but also the algorithm which created the store ids. It generated the store ids (which are actually just PathBufs) as absolute paths, so foo would become /foo. This was a bug.

The store ids are no longer absolute but relative. The store now adds its own path as prefix for each ID and everything just works now.

What's coming up

Lets have a look what might be merged in the next 14 days.

First of all, there is, of course, the hook system. It is completed, though it is neiter reviewed nor merged, yet. I guess it might get some more tweaks but it will get merged with rather minor fixes instead of big changes. I'm really looking forward to get this into imag!

The second thing is: There are a lot of pending PRs which are for libraries. I want to introduce the linking library, a “notes” library which will be the foundation for the “notes” module, tagging filters in libimagentryfilter and libimagentrymarkup which is a utility to translate entry markup content into HTML (for viewing with a browser). All these libraries (besides libimagentrymarkup) are ready for merging as far as I can see, though there was no review, yet.

I also want to add a store feature so one can move an entry (Store::move() and Store::save_as() will be introduced for this). The interface is ready and submitted as PR, though there is no implementation yet.

There are some modules pending as well: A imag view utility for viewing content (mainly a frontend to libimagentrymarkup, though it will also make use of things from libimagentryfilter and possibly others as well), a imag tag utility to modify tags of arbitrary entries and imag link for linking arbitrary entries. There's also a pending PR for imag counter which will add a simple counter module to the imag suite.

All these modules are waiting for the libraries described above to be merged.

imag counter

The imag counter module will be a rather simple module. It supports creating, deleting and listing counters and of course incrementing, decrementing, setting and re-setting them.

This is planned for the first release.

After that, I want to enhance the functionality of the library and provide several counting mechanisms (and composability so other modules can use the library to use the imag counter functionality in arbitrary ways). I want to be able to count and get a graph which shows on which dates (or times) I did the most incrementing-steps, so the counter needs to save the dates it was incremented.

All these things are rather trivial to implement, though I want to make the interface as generic as possible (and provide some simple things like described above), which makes this task rather non-trivial. That's why this is planned for later and not implemented in the first step.

github pages

Another problem I'm working on is github pages. I want to upload the imag crate documentation to github pages, though it is not that simple as the repository does not contain only one crate but several.

I also want to include the code-coverage statistics later on, though this would be another step and would be done in a seperate PR later on.

Readme update

The README.md file of the repository will get an update in the next few days, so it will list a lot more information on what the repository contains and what to expect from the project.

Random thoughts

Lets see... We have some awesome PRs merged in the last 14 days, there are more awesome PRs in the pipeline and about to be merged... so awesomeness everywhere, isn't it?

Well, yes. I really want to get the libraries merged into the master branch. After that and after the hook system is merged, I can implement modules for imag and I'm also able to implement hooks for imag. And I can develop them in parallel, as they do not depend on each other. This is really important to me, as I do not get blocked from my own PRs!

Let's hope we get this stuff in! Maybe this weekend, as I have a hacking session with some of my friends planned for tomorrow, so lets see... maybe I can report some major progress in 14 days!

tags: #linux #open source #programming #rust #software #tools #imag

This is the first part of a series of Blog posts I want to release every 14 days or so on what is currently brewing in the imag (Pull-Request-)Queue.

I hope I can release one article for this series every 14 days – read: I hope there is enough progress so I can write something on it every 14 days.

Lately there is not that much progress, which mainly is caused by me awaiting review by others. I don't know why, but it is not happening and pinging them does not help that much.

Lets see what is floating around in the Pull-request-Queue of the repository.

#130 – “Doc mod”

This PR is already over 1 month old. It is meant to document features the imag modules might have. Something like “The calendar should be able to {add, remove, list} events” and so on.

At this moment, there are a whole bunch of sub-tasks for this, one for each module I want to introduce to imag. For the lazy, here is a list with short explanations attached:

  • bookmarks
  • borrow-tracker (if you give something to someone, this tracker can be used to remind you after N days that you should ask to get your stuff back. Should use the calendar and the contacts modules)
  • calendar (via external tools, E.G.: vdirsyncer)
  • contacts (via external tools, E.G.: vdirsyncer)
  • counter (simply for counting things)
  • cue cards (If you want to learn things and you like using cue cards, this module might help you. Attaching wiki entries or external references will be made easy via imag, though main focus is on cue-card organisation)
  • dairy (This is rather simple, right? If the hook system works, this will be encrypted by default)
  • images (via extenal tools, E.G.: git-annex)
  • ledger (via external tools, E.G.: ledger-cli)
  • mails (via extenal tools, E.G.: mutt, offlineimap, msmtp)
  • movies (via extenal tools, E.G.: git-annex)
  • music (via extenal tools, E.G.: git-annex)
  • news (via external tools, E.G.: newsbeuter)
  • notes (simple note-taking tool. Will be used by a lot of other modules to attach notes to your stuff)
  • password (via external tools, E.G.: password-store)
  • personal library (To track references and PDFs and handle bibtex (for example))
  • todo (via external tools, E.G.: taskwarrior)
  • weather (for saving the weather in your store, forecasts and such)
  • wiki

The above are all high-level modules. Some more modules will be low-level modules, which are there to access the store in a module-generic way and which could be used by the user to modify single or multiple store entries, though the high-level modules should do this on their own. These modules are “just” interfaces to the respective libraries.

* link helper (CLI for the linking library) * tag helper (CLI for the tagging library) * git helper (not sure on this one, though) * content-to-website (CLI for the entrymarkup library, markup converter for viewing things in the browser) * store-integrity check (find dead links, etc. Not sure on this one either)

I guess this issue will be there for a long time.

#167 / #171 – Linking

167 is a PR for the documentation which defines how linking should work. 171 is the implementation of the things described in 167.

The PR (171) is ready and waits for review. The linking library tries to ensure that the entries are two-way linked and supports both internal and external linking.

All modules should use this library for linking later on, to ensure the linking works.

#174 – Tagging

174 is the PR for the tagging library. It implements a helper for adding, removing and setting tags in an entry. This was added because tagging is rather essential for a lot of modules (bookmarks, counter, cue cards, images, ledger, mails, movies, music, news, notes, personal library, todo, wiki). Some of these modules might be implemented using external tools (“todo” with taskwarrior for example). These modules might re-implement parts of the tagging mechanism of the tool to support imag. I'm not yet sure how to handle these external tools and discussion is clearly needed, of course.

The PR also includes a helper for building a commandline argument or commandline subcommand for specifying tagging actions. These can be used to build up similar commandline interfaces and parse values of them via the tagging library, so we have a clean UI and a single point of failure for tag handling.

#191 – Markup processing

Here I implement a high-level library which helps converting entry-content via different markup-processors into HTML pages.

This library shall be used to show content from the store to the user in a convenient way by translating content which was written in a markup language into HTML. The user of the library does not know what markup processor is used and she shouldn't care. The library gets an entry and spits out a chunk of HTML which can be shown to the user than.

#192 – “note” module

This is the first “high level module”. The PR implements the “note” module, though it is currently on hold, as I have to wait for three other PRs to be merged (171, 174, 191 – linking library, tagging library and markup library).

#193 – “counter” module

This is another “high level module” – The “counter” module. It also waits on three more PRs to be merged (171, 174, 191).

#200 – Hooks for the store

Well this is the most complex PR right now. It implements the Hook system for the store. The hook system offers different types of hooks, which are realized via traits, so everyone is able to implement hooks for the store.

These types of hooks are supported:

  • post-create hooks
  • post-delete hooks
  • post-read hooks
  • post-retrieve hooks
  • post-update hooks
  • pre-create hooks
  • pre-delete hooks
  • pre-read hooks
  • pre-retrieve hooks
  • pre-update hooks

Parallel execution of Hooks, although there might be RW-Hooks

The challenge here was to implement the traits for the hook types in a generic way, so each hook is able to either access the data it gets in a read-only wait or aquire write access on the data. This is currenly implemented via an “Accessor” type, which is an enumeration over either a read-only accessor or a read-write accessor (this is not integrated in the PR at the time of writing, as this is highly experimental). With this system in place, the hook system of the store is able to decide whether hooks should be ran in parallel or not. For example, if there are only Read-Access-Hooks in the list of hooks which shall be executed for a, say, create() call, the hook system is able to execute these hooks in parallel.

Not all types of hooks provide this feature, as some of the hooks are not allowed to alter data. For example a pre-read hook only gets the StoreId which shall be read, there is no point in altering the ID on read(). A post-read hook might alter an entry and therefor has to be able to provide an “Accessor” object, which then can be called by the store RO or RW.

Sorting hooks?

Another thing in my queue for this PR is the sorting mechanism for the hooks. One hook might be executed before another or after another. Hooks do not know of eachother – so you cannot compare them. So I had to “invent” a system how each hook can tell the store when it wants to be executed. There are five possible values:

  • First
  • earlier
  • doesn't matter
  • later
  • Last

As far as I can see, this is not yet optimal and I have to come up with something better. That's the point why this stuff is not yet included in the PR 200 – it is not yet perfect IMO.

What the hook system may be used for

The hook system is meant to move functionality of the store out of the store library. We argued on having the store version controlled via git. One of us doesn't want that per default, as “It would pull in git as dependency” (I wouldn't mind). Having a hook system would enable us to provide version control via git or whatever other tool the user uses, without integrating the functionality in the store.

Encryption of store entries via gnupg would be easy, as the entry gets decrypted on reading into memory and encrypted on writing to disk. The store and the modules wouldn't even notice.

Consistency checking, integrity checking for linking, tagging or whatever else would be rather easy. All these things could be implemented without altering the core of the store library.

Some of these functionality would still be implemented in the core distribution of imag, though. I would really like to add encryption and version control support as libraries to the distribution and register the hooks for them in the store by default. This does not alter the store library, but the runtime library (which initializes the internal store object). Version control would be enabled by default. I'm not sure whether encryption should be enabled by default, though I doubt anyone would argue against encryption, as all the data which is in the store is rather personal (thus personal information management suite).

More PRs

There are some more PRs floating around in the repository right now. None of them is that large, just little enhancements and nice-to-have things, like colored output for the log, error tracing in color, a ismatch!() macro for less noise in code blocks like this:

if !foo.iter().all(|f| match f { &SomeTypeWithValues(_) => true, _ => false }) {
  // ...
}

// but simply:

if !foo.iter().all(|f| is_match(f, &SomeTypeWithValues(_))) {
  // ...
}

Lately merged

Some things were merged in the last five days, though I'd love to see more things merged. But as said, there were no reviews, so I couldn't merge things.

  • The runtime library removed support for short versions of “—rtp” and “—config”, as the modules migth need “-r” or “-c” (#205)
  • Regex are now compiled once, even if they are called in a loop - lazy_static!() helps with this (#206)
  • clap and regex crate dependencies were updated (#207, #214/#215)
  • Some cleanup was done and unsused imports were removed in libimagentrymarkup (#218)

Random thoughts

I'm currently thinking about the problem on how to integrate external tools without breaking user experence.

For example I use taskwarrior and I really love it. I really want to integrate it into imag, though I still want to be able to use the taskwarrior binary from the commandline as I'm used to. But after altering things in taskwarrior, they should be available in imag as well, without some futher hassle.

As taskwarrior stores its contents in JSON files, I could simply parse them and use this data, though I don't like that idea that much because if the format changes, I have to adapt my software. Also, linking to this content is rather complex, as taskwarrior seems to store everything in one big file rather than in one-file-one-task. Linking to tasks could become a major pain here.

The same goes for ledger-cli. ledger-cli supports putting each transaction into one seperate file, though the user has to cope with this when using ledger from the commandline. At least as far as I can tell.

None of these tools do export an API. Even if they'd export one, I don't think they would allow to re-implement the storage backend, so I could build an interface to my store implementation. And of course re-implementing their functionality is really not what I want to do.

taskwarrior has a hook system since 2.4.0 which could fit into my needs. ledger has not.

tags: #linux #open source #programming #rust #software #tools #imag

“imag-store” was merged today! This is the first commandline tool for imag – the commandline personal information management suite!

Okay, well, it is just the bare interface to the store and has no special features, as it is meant for debugging purposes only. Though, it is the first binary and we can finally see some progress at the CLI front! Yay!

What's next?

I'm writing some libraries at the moment, one for tagging entries and one as convenience layer for linking entries to internal and external resources (but this isn't fully specified yet). I'm also writing some utilities for entry headers, so you can query them from the CLI with a DSL (as well as in a module, but the query is interpreted at runtime and isn't build at compiletime which is way to complicated for me at this moment). I'm also trying to get the git layer in the store working, but we have a discussion about whether it makes sense to implement this non-optional or not. I'm strongly for integrating it, as it makes the store code way simpler and of course it is a big point of the store to have version information available.

And modules?

Nothing keeps you from writing one. Really, all the things are already there and implement. One last thing you might want to wait for is the implementation of the retrieve_copy() method of the store which is not completed yet. But all the other things are there and usable (tests are there, but I really would be careful – I do not consider the store “stable” yet)!

So what are the concrete plans?

  1. Get the libraries implemented
  2. Test the libraries
  3. Start writing modules, the first modules will be the “notes” module and the “bookmarks” module, I guess.
  4. More modules
  5. More modules
  6. More modules

tags: #linux #open source #programming #rust #software #tools #imag