When we designed the imag core functionality – the imag “Store” – we introduced a type called StoreId.

It turned out that the way we designed this type was a big mistake and turned out to be the first big imag design flaw by now. With this post I try to wrap my head around possible solutions to this flaw.

What is this about?

So we wrote the StoreId type. It is a wrapper around std::path::PathBuf, so it is basically a path. We wanted it to be the path to a store entry, but it should be absolute to the store root itself. So a valid StoreId object could be "note/car" – it would uniquely identify a file in the store. We made the StoreId also contain a version part, to be able to write imag modules which could check their data for compatibility with the current version of the library.

The Problem is that we just wrapped the path type. A StoreId can be created from any PathBuf and even from any String – which clearly is not the way it is supposed to be.

We didn't use the type system enough. We should have designed this type more carefully. Lessons learned, I would say.

I know that I should write I messed up, but as the StoreId type was also designed by my friend Marcel, I use the term we.

How to clean this up?

Well, this is the hard part. As we do not have a release yet, we do not have to be backwards compatible, so there is no problem with that.

Anyways, we need to rewrite a lot of code to fix this design flaw. As imag gets more momentum in the rust community, this gets more important. This design flaw is actually blocking some other things, as we cannot rely on the properties of the StoreId as we should be able to.

One idea is to rip out the StoreId type completely and redefine it. In my opinion, a StoreId should only be constructed by the Store, upon request by the store user. That means that the user should be able to store.new_id("module", "path/to/entry") which then creates a new StoreId that points to $STORE/module/path/to/entry. That would be a clean way. The StoreId type then may contain functionality to create, retrieve and get and maybe even delete the entry it points to (so it contains a reference to the actual Store object internally).

I'm still not sure about the details. As said, I try to wrap my head around this problem with this very article.

tags: #linux #open source #programming #rust #software #tools #imag

Ten days ago I published another crate on crates.io - the 'filters' crate.

Its source code was partially extracted from imag. In this article I want to show you what you can do with it as well as list some things that should be improved.

How to Filter<N>

First of all, Filter is a generic trait. It can be implemented for a wide range of things, of course, but is implemented for all T: Fn(&I) -> bool. It requires one function: Filter::filter(&N) -> bool to be implemented. This function should hold the predicate definition. So you can do something like this:

use filters::filter::Filter;
struct EqTo {
    pub i: usize,
}

impl Filter<usize> for EqTo {
    fn filter(&self, n: &usize) -> bool {
        self.i == *n
    }
}

fn filter_with_eqto() {
    let eq = EqTo { i: 0 };
    assert_eq!(eq.filter(&0),  true);
    assert_eq!(eq.filter(&1),  false);
    assert_eq!(eq.filter(&17), false);
    assert_eq!(eq.filter(&42), false);
}

in your codebase. Of course, that is a lot of boilerplate code for simply filtering something for beeing equal to a number, right? That's why we implemented it for all T: Fn(&I) -> bool – you can boil down the upper example to this:

fn filter_with_eqto() {
    let eq = |&a: &usize| *a == 0;
    assert_eq!(eq.filter(&0),  true);
    assert_eq!(eq.filter(&1),  false);
    assert_eq!(eq.filter(&17), false);
    assert_eq!(eq.filter(&42), false);
}

And/or/not now?

This does not yield any code abstraction improvements yet. But the Filter trait has also has some more functions which are already implemented for you: and, or, not and so on.

This is where the other types from the crate come into play: There are types available which implement logical operations using the Filter trait. And the functions from the trait itself use them so they are convenient:

fn filter_inside_range() {
    let zero = |&a: &usize| *a > 0;
    let hund = |&a: &usize| *a < 100;

    let f = zero.and(hund);

    assert_eq!(f.filter(&0),  false);
    assert_eq!(f.filter(&1),  true);
    assert_eq!(f.filter(&17), true);
    assert_eq!(f.filter(&100), false);
}

And of course you can chain them:

fn filter_inside_range() {
    let zero = |&a: &usize| *a > 0;
    let hund = |&a: &usize| *a < 100;

    let f = zero.and(hund.not()).not().or(true);

    assert_eq!(f.filter(&0),  true);
    assert_eq!(f.filter(&1),  true);
    assert_eq!(f.filter(&17), true);
    assert_eq!(f.filter(&100), true);
}

As shown above, we also implement Into<Bool> for bool so you can easily provide default values for your filters.

What this crate is for

This crate was designed for imag, but should prove helpful elsewhere, too. It was written to be able to construct complex filters based on user input. For example if your application has a commandline interface which allows the user to specify predicates and logical operators, you can use this crate to build these filters and connect them.

That is also exactly the usecase for imag, by the way.

Issues

There are two remaining things I want to implement in the codebase. The crate is usable as of the time of writing, but I also want to do the following things:

• Implement BitAnd, BitOr, ... etc for all filters. With this, one could do (written in short now) (|a| a > 1) & (|b| b < 5) – using bitwise operators to make these things even nicer.
• Provide a possibility to use a Filter as function, so one can do myvec.iter().filter(foo) instead of myvec.iter().filter(|x| foo.filter(x))

These issues are posted in the github repository of the crate and of course you are welcome to help out.

tags: #open source #programming #rust #software #imag

Another 14 days – a lot has happened in the repository of the imag codebase, the commandline personal information management suite written in Rust.

Read a summary of the last two weeks here, in the 13th iteration on what's coming up in imag.

The past

Lets see what was done in the last 14 days.

Website, IRC, mailinglist!

First: The important things!

We have a website now! Meet us at imag-pim.org. We also have a git mirror there.

So if you want to contribute to imag but you do not have a github account or you do not want to contribute via github, you can ask me for a repository on the git repository on our website and contribute via mail or via this repository. I will push your patches to github PRs anyways, because of our CI server, but you do not have to interact with github at all. For details, read our CONTRIBUTING file.

We have an IRC channel now! Since 1st of August you can join irc.freenode.net/#imag and talk to us/me directly!

The channel has a bot (nick “imagned”) running which can be used to log discussions. It is a sopel bot, see the page on the “meetings” module for documentation how discussions can be logged.

We have a mailing list now! Since 2nd of August you can join our mailinglist and talk to the imag community via it. Of course, there are few mails posted yet, as this list is really new.

We accept patch mails via this mailinglist as well! Make sure you've read the CONTRIBUTING guide before submitting patches.

Numbers!

We reached the 100 stars on github! Awesome! Thank you all for this! We are reached the 2,500 commits and we reached 400 closed PRs!

New contributors and integration of the first external tool!

Besides that, we have merged #383, which was a huge one. It implements libimagtodo and imag-todo, using taskwarrior hooks to store references to taskwarrior tasks in the imag store. This allows us to refer to tasks created with taskwarrior from within imag. imag-todo is therefor the first imag module using an external tool to implement the PIM aspect it covers. This is huge! I hope this is the first step into the direction of multiple backends as well.

The PR was created as semester project at Hochschule Furtwangen University and therefor contains commits from several contributors (in alphabetical order):

• Mario Krehl (who already has some commits in imag, though)
• Roman Schemenau
• Sascha Schwenteck
• Yasemin Dogan

I did some cleanup in this PR as well as some restructuring. So it was joint work. Because of this, the shortlog tells us the following:

• 52 commits from Mario
• 51 commits from me
• 6 commits from Yasemin
• 4 commits from Sascha
• 2 commits from Roman

Thanks a lot for your contributions!

Other contributors were:

• Gavin Thomas Claugus
• Kai Sickeler

Thank you a lot, too! I also want to thank Julian Ganz and Pascal Hertleif for contributing ideas and participating in discussions on issues.

PRs merged/closed in the last 14 days

We had a lot of small PRs in the last 14 days, fixing one or two things. This is truely awesome, but the list for PRs is even longer than from last week. I will only add notes to the bigger ones here, I guess I if this trend continues I will start to list only the noteworthy PRs in the future.

But here we go, with the full list of PRs and some of them explained or with an additional note.

• 383 (explained above)
• 594 added an utility to print debugging information while map()ing over Results or Options.
• 595
• 597
• 598
• 599
• 600
• 601 updated the user documentation a bit.
• 604
• 605
• 607
• 608
• 609
• 610 added an EditorView in the libimagentryview library, so one can now view entries with the $EDITOR.
• 611 fixed a serious locking bug in the libimagstore library.
• 612 added support for --no-color output in libimagrt (so all modules get it automatically).
• 613
• 614 added a TableLister type for libimagentrylist – not tested yet.
• 616
• 617
• 620
• 621
• 622
• 623
• 630
• 631
• 633
• 634
• 636
• 640
• 641 fixed our travis setup and shortened build times from up to 50 minutes down to about 30 minutes.
• 642
• 643 outsourced editor-calling code from libimagrt into the new library libimagentryedit.

PRs opened in the last 14 days and not yet closed

• 596
• 603
• 606
• 624
• 635
• 638
• 639
• 645 adds custom hash support for libimagref.

Issues opened and already closed

These issues were opened and are already closed:

• 619
• 626
• 627
• 632

The future

Lets see what the future will bring us.

Issues opened and not yet closed

First of all we have a bunch of new issues which are not yet closed.

• 602 asks whether we can remove the version part of the StoreId.
• 618
• 628
• 629 asks for an explicit subcommand with imag-todo to import the task export exported data from taskwarrior.
• 637 asks for a feature in libimagref so we can only hash the first N bytes to speed up both hashing and refinding of entries (which is untested btw).
• 644

Issues reopened

I reopened issues #2, #3, #4 and #5, which ask for a contacts-, calendar-, mail- and wiki-module respectively. Unfortunately, there is no vcard/icalendar crate for rust available. There is vobject, but this is a very basic abstraction over the filetypes. I would need something more complex and powerful, which has a lot capabilities and hides the file format in nice abstractions. For example, if I want to check whether two calendar entries collide somehow, the library should be able to check whether the entries are recurring and whether they collide some time in future ... or the contact library should offer a simple interface for querying numbers, mail adresses, adresses, names and so on. With the vobject crate I would need to implement all these things on my own – I really don't want to do that right now!

Also, I would need a library to read mails from a Maildir for the mail module. There is mailparse – I'm not sure whether it fits my needs, though.

For the wiki part I don't need anything besides markdown parsing capabilities, and I guess I already have them. I would implement the wiki module with the libimagref library, which can then be used to ref to existing wikis such as vimwiki. I'm not sure whether this is the proper way to do this, though.

Other things

Vacation! I will be on vacation from Saturday August 13th until Tuesday, August 23th. I will be at the Summerbreeze Heavy Metal Festival (feel free to meet me there!) from Tuesday, August 16th until Sunday, August 21th, so there will be no commits in this timerange (at least not from me) and of course no merges.

Because of that I'm pretty sure that the next iteration on imag will be a short one – but well, I guess I deserve a break from my bachelors thesis which is almost finished.

And also because of this, I do not have any plans for the next 14 days. Lets see what comes up... lets hope something great! Maybe some new contributors...

tags: #linux #open source #programming #rust #software #tools #imag

Yesterday I got asked what the usecases for imag are. I tried to explain it and afterwards thought about it the whole night.

I already wrote about the purpose of imag, but I guess some usecases would explain even better what imag is for.

So now I try to explain three usecases for the imag commandline personal information management suite.

Please keep in mind that these are the usecases how I would describe them as final goal of imag. None of the following is implemented and all of this is music from the future! But that's the ultimate destination of the project.

Go shopping for me.

You get an Email from Sally. A hook script informed imag, so it already knows about the mail, has fetched the contact information from the email header and linked the email to the vcard of Sally. In the mail, Sally asks you if you want to cook with her and if yes, you should buy eggs. You type in a shortcut to call imag. You tell imag that you want to create a new shopping-list. imag creates a new shopping list. Because you called it from a mail, it asks you to link this mail. After you created the shopping list, imag creates a task in your todo tool for you. It asks you whether you want to create a calendar entry for the shopping list.

Later you add things to the shopping list. You add potatoes and sweet chestnuts. imag recognizes the change in the shopping list. It notices that you had similar shopping lists that contained Eggs, potatoes and sweet chestnuts and that they were linked to a recipe. Imag asks you whether you want to link that recipe to the current shopping list.

After you went to the mall, you type the cash slip into your ledger. imag recognizes that you bought stuff you had on a shopping list that isn't marked as done yet and therefor asks you whether the shopping list can be marked as done. You mark the shopping list as done. imag links the ledger entry with the shopping list. imag asks you whether it should mark the linked todo entry as done for you. Yes, you want to close this todo entry. Because you went shopping before the calendar entry, it asks whether it should delete the calendar entry for you. You delete the entry. imag recognizes that you linked an email – but no response for that email exists. So it asks you whether you want to reply to Sally. You reply to the mail and tell Sally that you buyed the things and you'd love to come over and cook together.

You query imag for that recipe imag told you about and add it to the mail, asking Sally whether this would be appropriate. imag has already linked your response to the original mail from Sally. imag also linked the recipe to your response. imag recognizes that there are notes attached to the recipe and asks you whether they should be in the mail as well.

You type in another shortcut to create a new calendar entry “Go to Sallys, cook with her”. imag links that calendar entry. imag recognizes “Sally” in the subject of the calendar entry and asks which Sally from your contacts should be linked to the calendar entry.

New contacts and a party

You were at a party and met Amelie. You add her contact information to your contact application. imag knows that contact now, too. You tell imag that you met Amelie at the calendar entry for yesterday evening. imag links the contact with the calendar entry from yesterday evening. Because the calendar entry is linked with a location and the website of the club, imag knows that these pieces of data are in relationship with eachother.

A few weeks later you're typing an email to Dave, asking whether he would come with you to the club. You query imag for the website of the club. imag links that email with your bookmark of the website of the club.

Your friend replies yes (imag links the reply with your mail), so you add a calendar entry. imag recognizes that you linked the website of the club and a calendar entry with mails between you and Dave. It asks you whether it should show you other contacts that are somehow in relation with calendar entries that are related to the bookmark of the club. You find out that you met Amelie some weeks ago at this club. You invite her, too.

Further, imag tells you that it knows from your ledger that you spend around 50 bucks last time you had an event that was linked with that club (on cola and water, of course). It also knows that you have about 20 bucks in your pocket right now. imag asks you whether you want to create a task “Go to bank”. Yes, you want that. imag finds out that you have time tomorrow at 4pm and asks you whether you want to create a calendar entry for this at that time. Yes, you want that. Because the other calendar entries from tomorrow are linked to locations downtown, imag infers that you are in this location and finds the best bank nearby for you, asking you whether that location should be linked to the calendar entry.

Business, Business!

You get an email from your boss, she tells you that you have to fly to another country tomorrow, to attend an important meeting with a customer. She sends you a icalendar file containing information about the flight. imag already knows that this calendar entry has to go to your business calendar, because the mail was from a contact from your business contacts.

You add the calendar entry via a shortcut from your mail client to your business calendar. imag recognizes that this is a business trip and asks you whether it should search for packing lists. It could also create a todo for you “Pack things for business trip” and link the packing list to it. Hell yeah, you want that!

Because all your flights are linked to a location that is about 50 miles away from your home, imag asks you whether you want to create another calendar entry “Drive to Airport” right before the flight. Of course you want that. Because there is another calendar entry which collides with the trip to the airport, imag asks you whether you want to create an email to the contact linked to that calendar entry. imag has already figured out an alternative date for the meeting you have to postpone.

---

You might call me crazy, but yes, that is exactly the way imag should work in the end. Of course, this is far far away in the future as of now, but I do not think that it is unachievable!

How the interaction with imag works in detail is not described here, of course. I don't know it! It will grow with the project. Maybe all the above will be done via commandline calls, maybe there will be some fancy curses UI or maybe there will even be a web frontend (though this is less likely).

imag is a tool for nerds, from nerds. So it will require some scripting and fiddling to get imag integrated in all the tools. imag tries to provide a convenient interface for all these things, but not a out-of-the-box solution (maybe in 10+ years, but it is not a short- or mid-term goal).

I hope that these use cases explained some things and helped you understanding the purpose of imag.

tags: #linux #open source #programming #rust #software #tools #imag

imag just got a website, a mailinglist, a git repository setup and an IRC channel.

So I wanted to set up a website for imag for months already, but I finally got to it. I actually developed the website offline and it was almost done, and now it is online.

I wrote the website using nanoc, a static site generator. I used nanoc before and I also contributed some code to nanoc, so I already knew what to do and how to do it. I wrote a minimal theme for the website, I wanted to have it plain-text-ish, as imag itself is a plain text tool.

I managed to register an IRC channel on the freenode irc network, which is awesome. travis was set up to post to this IRC channel if a build succeeds or fails which is really convenient as well.

Of course we also have a mailinglist now. One can register there to contribute patches via mail and to ask questions. Of course there's not that much on the mailinglist yet, as we do not have a community around imag, yet. Anyways, there's the possibility to build one now, which is awesome, I guess!

Background

So what's running in the background here?

I registered a space at uberspace where also this very website is hosted, set up a gitolite and a cgit webfrontend for the git repositories.

The mailinglist is run by ezmlm, which was written by djb and is very well documented how to setup on an uberspace.

The domain was registered on united-domains.de.

Costs

Well, I pay these things from my own money (I can make some money this summer working at my university, so that's not a big problem).

Currently, I pay 19 Euro per Year for the domain and 2,50 Euro per Month for the Uberspace, but I will increase this as soon as there are more contributors on the git hosting or on the mailinglist (as soon as my setup causes actual workload on their servers) or as soon as I have a job, whatever comes first.

So that makes it 49 Euro per year with the current setup, which is affordable for me. As soon as I increase the monthly fee for uberspace (I will go to 5 Euro if I make my own money and no contributors and more if there are contributors), this will cost me 100 Euro per year if I give uberspace 6.75 Euro per month. Still not much, I guess.

As soon as I have to pay more than 100 Euro per year I guess I will add a “support this project” button on the website ... or something like this. Well, we'll see...

tags: #linux #open source #programming #rust #software #tools #imag #git #mailinglists #network #social

Another 14 days in imag, the personal information management suite for the commandline – awesome!

We got a huge bunch of PRs merged in this period, which is awesome! Read more about it in this 12th iteration on what's coming up in imag!

The past

We got PRs of three contributors merged in the last 14 days. Thank you very much, Kai Sickeler, Gavin Thomas Claugus and LeRoyce Pearson!

Also, we

• hit the 80 stars-mark on 16th of July
• hit the 90 stars-mark on 25th of July
• hit the 2200 commits-mark on 25th of July

This is so awesome!

PRs merged in the last 14 days

As the number of PRs was rather high in the last 14 days, I just explain the most important ones here, leaving the others uncommented. If one wants to dig deeper, the PR description or title should suffice.

• 522 adds the implementation for imag-bookmark – the frontend for libimagbookmark, which is implemented in #521. 301 was closed.
• 514 was closed for 569, which superceeded it.
• 542
• 543
• 544
• 545
• 546 adds more levels (more details) in the hook execution error wrapping
• 547
• 548 fixes the MutableHookDataAccessor to also execute StoreIdAccessors
• 549
• 535 added unit support for imag-counter.
• 550 fixes the example imagrc file.
• 551
• 552 removes some trait bounds from the hook traits, as they do not have to be Sync.
• 553 adds non-aborting support for hook errors. This means that a hook can return with an error which does not abort the store action it was called for. For example if a Store::create() call invokes a hook which fails somehow, the error can be marked to not abort the Store::create() call. By default, hook execution errors are aborting.
• 554
• 555
• 557 merged commits so the store does not get created implicitely anymore.
• 558
• 559 refactored imag-view.
• 561 adds a feature to the store which can be opt-in'ed at compiletime which gives the possibility to verify the store contents. This is implemented in the imag-store binary, which now has a subcommand to verify the store.
• 562
• 563
• 564 fixed some issues with the imag binary behaviour which was not very convenient at the time.
• 567
• 568
• 570 (closed unmerged, superceeded by 571)
• 571 added support for config override the configuration via the commandline.
• 573
• 575
• 578, 579, 580, 581, 582, 583 and 584 fixed the missing deny() blocks – warnings will now be treated as errors and abort compilation.
• 586
• 587
• 588 refactored libimagcounter a bit.
• 589 added a meta-crate for building the documentation of all libimag* crates in one step.
• 592 made travis-ci silent.

PRs opened in the last 14 days and not yet closed

• 569
• 593

The future

Lets see what the future brings us...

Issues opened and already closed

The following issues were opened in the last 14 days but are already closed.

• 556 asked for a rewrite of imag-view.
• 560 mentioned an issue with the info-output which wasn't shown if --verbose was passed.
• 565 requested support for overriding the configuration via the commandline.
• 572 mentions a bug that imag-view overrides --version in an unexpected way.
• 574 mentions an inconvenience with the location of the imagrc file.
• 585 was mentioned by Kai – some small enhancements can be made to libimagcounter.

Issues opened and not yet closed

The following issues were opened in the last 14 days. I might have missed some in this list, but here are the important ones:

• 539 is a question whether we want the clap feature “suggestions” in imag and whether we want to create a wrapper crate for this or use the more noisy variant and enable it in every single Cargo.toml
• 541 is a RFC for a imag-repl command. Would be a nice-to-have, I guess. I'm not sure about this, though.
• 566 requests a switch to clap for the imag binary.
• 576 mentions that we use unwrap() quite to often.
• 577 mentions that we might implement a curses frontend for imag-view.
• 590 asks for libimagnotification which should use notify-rust to create a notification utility library.
• 591 asks for an update of the README.

Other things

I'm still working on the git backend implementation and I really hope I can get some progress. Right now I'm having a weird panic because of an RW-Lock. I don't know why, though, and I really hope I can solve this quickly. I fail finding/seeing the issue, though.

tags: #linux #open source #programming #rust #software #tools #imag

We cracked the 70-stars mark on github! Wow. We reached the 2,000-commits-mark. Wow. We also reached the 500 PR/Issue mark. Another wow.

But there happened a lot more things in the last 14 days in imag, the personal information management suite for the commandline. Read it up here.

The past

Wow. 14 days can be so damn short. We got a few things done in the last 14 days, but first, I want to thank all contributors:

• mario-kr who did a semester project at his university with some fellow students. He submitted a pull request to implement libimagtodo, which is a todo module for imag, currently using task-hookrs as backend for talking to taskwarrior. This is really great!
• mario-kr also submitted some more PRs:
  • A bugfix for the bin/imag binary
  • A feature for task-hookrs to import a task from taskwarrior from a single JSON object.
  • A lot of suggestions and comments in issues what is wrong with task-hookrs.
• jsirois opened two PRs after reading my this-week-in-rust entries where I submitted two issues which could be done by people wanting to learn some Rust: 531 and 532 – absolutely awesome!
• I also want to thank impo who submitted a PR to build the docs to a manpage. This reminds me that I really should work on the docs. Either way, a big “Thank you!” to you, impo!

Wow, so much contributions! Awesome, keep it up, guys! Lets take a look on the merged pull requests.

PRs merged

The following PRs were merged in the last 14 days:

• 468 added libimagref - a library to create references to files on the filesystem. Something like pointers to external data. This can be used to reference files outside of the store, track them by content (SHA1), their permissions, their location and so on. This will also be used to implement a list of other tools, such as a wiki module, a movie-, music-, images-module and so on. There are a lot of possibilities here.
• 481 implements a frontend for libimagref for the commandline, so a user can use libimagref directly to create filesystem references.
• 487 added an implementation of Display for StoreId. You can now println!("{}", id) – awesome!
• 489 refactored the internals of the store a bit, by moving a function from Store to StoreId. This shouldn't change the use of libimagstore at all.
• 496 made the CoreLister-Type from libimagentrylist more generic. You should be able to pass not only closures, but also normal functions now.
• 500 contained a bug-fix for 468
• 503 fixed a flaw in the Display implementation for StoreId
• 513 adds a configuration option which makes it possible to deny altering hooks per aspect in the store hook execution
• 517 fixed a bug in libimaginteraction which caused the ask_bool() (and possibly other) function to loop endlessly
• 518 removes a unneeded dependency from imag-view
• 519 puts the out directory in the gitignore file
• 521 adds the implementation for libimagbookmark – the bookmark feature of imag (the library part).
• 525 updates the documentation. Before, the documentation was a technical document. I do not believe that this will bring us any advantages anymore. This PR rewrites the document to be a user documentation.
• 528 updates all the crate versions from “0.1.0” to “0.2.0”, as we currently develop imag in this newer version.
• 531 removes some unused code and therefor cleans up the warnings the compiler gives us about the unused code. This PR was later superceded by 536 and 537.
• 532 adds a fold_result utility function for all iterators and therefor closes
• 533 fixed a header-field naming error in libimagref.
• 534 introduces a (hopefully) temporary fix for libimagentrylink. The check, whether a StoreId object points to an external link store object or not was buggy as it assumed the PathBuf inside the StoreId to be absolute to the Store root.
• 536 superceded 531
• 537 superceded 531
• 538 refactored some code in the store to remove duplication.

PRs about to be merged

And here are some PRs which are work in progress and might be merged in the next 14 days. These were opened in the last 14 days and I hope to close them (merged or not merged) fast... some of them seem to be pointless (the ones on the store refactoring for testing are kind of duplicated as I tried different approaches) so they might get closed unmerged.

• 486 implements a git store hook, which then puts the store into a git repository and tracks changes within the store. I'm not sure whether I will be able to merge this, as I still have problems getting the git2 crate building, as zlib is somehow missing, though I added it to my nixos setup... we will see...
• 497 implements an iterator type for libimagref, so we can iterate over references in a nice way.
• 498 uses 497 in imag-ref (481) to list references if the user requests this.
• 514 adds an abstraction layer for the filesystem which should simplify store testing option which makes it possible to deny altering hooks per aspect in the store hook execution
• 522 adds the implementation for imag-bookmark – the frontend for libimagbookmark, which is implemented in #521 - so this is basically the other half of the feature.
• 529 is an attempt to pass only store-absolute StoreId-instances to the Entry objects inside the store. This PR might be superceded by 530
• 530 tries to rewrite the Store internals to use another StoreId type (EntryStoreId) inside the Store implementation which is always absolute to the Store root and never relative to the filesystem root. 499, which is absolutely awesome.
• 535 introduces a feature for imag-counter, which gives you the possibility to add units to your counters (when creating them). So you can attach a meaning to the counters and everybody will know that this is not the count of your daily pushups but rather the number of jellybaby you've eaten!

The Future

So what will happen next? I opened quite a bunch of issues in the last 14 days, most of them are rather simple things to do!

Issues opened/I want to get done

We reached the 60-open-issues line as well in the last 14 days. This is not necessarily a bad thing, as these are not all bugs but also feature requests and so on. Here is a list of issues which were opened in the last two weeks. I hope I can close some of them within a few days (some of them might be already closed at the time of writing).

• 488 [open]
• 491 [closed]
• 493 [open]
• 494 [open]
• 495 [open]
• 499 [open]
• 501 [open]
• 505 [open]
• 506 [open]
• 507 [open]
• 508 [open]
• 509 [closed]
• 511 [open]
• 515 [closed]
• 516 [open]
• 520 [open]
• 523 [open]
• 524 [open]
• 526 [closed]
• 527 [open]

I'm not including a description of each issue here... this would yield this post waaaay to long actually.

Other things

Well, I am in the last phase of my bachelors thesis right now... so I would say: Plenty of time to work on imag, right?

I hope to get the bookmark implementation finished and the listing support for the reference utility merged. Besides that (and after the reference foo is merged) I would love to start implementing a backend for the diary or bookmark module by either using jrnl.sh or buku. Not because I use them as commandline tools (I'm not saying I wouldn't) but because they seem to be simple to adapt. Maybe I'm wrong... but well... lets see.

tags: #linux #open source #programming #rust #software #tools #imag

I just released the second version of my task-hookrs crate, a crate for interfacing with taskwarrior!

This release is release 0.2.0 – so nothing too serious by now, as we still prefix with 0., but anyways a few nice things were merged and it seems to work pretty well!

These features were added:

• Annotation support
• Optional values are no longer included in the output
• import_task() for importing a single task JSON object

Bugfixes and other things were also done:

• Dead code removed
• Documentation written and updated
• Links in documentation fixed
• Import errors for rust beta fixed

And the crate is already published on crates.io.

So go ahead and use it!

tags: #linux #open source #programming #rust #software #tools

imag has made little progress in the last 14 days, as I'm really focusing on my bachelors thesis and exams right now. Anyways, I got some neat things done. Read here what happened in the last 14 days.

The past

Yes, my thesis is my main focus right now. This does not mean that imag is stalled, indeed I got four merges done in the last 14 days, and some more are waiting.

Developers Certificate of Origin

I added a Developers Certificate of Origin in a pull request. This means, that all developers/contributors of imag agree to the terms listed in this certificate.

Within this pull request I also added some notes how to contribute to imag without having a github account. People are welcome to send me patches for imag via mail! They just have to acknowledge that I will integrate them by opening a PR on github myself. Also, they have to add a "Signed-off-by:" tag in their commit messages, I will not include patches send by mail without this line.

I think that shouldn't be an issue.

Result::map_err_into()

This was the second pull request. It was followed by two pull requests which moved the codebase to use the feature from this PR.

What it implements: An additional method for the Result type, so one can use the Result::map_err_into() method now to minify error handling code in a sane way.

Before this pull request, one had to write the following code over and over again, to wrap an error type into the module-specific error type:

some_operation_that_might_fail()
    .map_err(Box::new)
    .map_err(|e| ModuleError::new(ModuleErrorKind::FailingOperation, Some(e)))

This quickly became rather ugly. With the new feature applied, we can write this now in a really short and neat way. The following is equal to the code snippet above:

some_operation_that_might_fail()
    .map_err_into(ModuleErrorKind::FailingOperation)

With the map_err_into() method doing the heavy lifting of packing the error into a new Box and creating an error instance with the passed error kind.

As this was implemented in the libimagerror code generator macro, this is available for all crates which use the libimagerror macro generate_error_types automatically.

The PRs that followed that improvement removed a lot of boilerplate code:

4 files changed, 63 insertions(+), 113 deletions(-)

The Future

Lets see what's in the pipeline. Most of the following things are “I want to do this” things, and I won't be able to do them that soon, as the exams are coming up. I really hope I can get some progress, still.

References

The libimagref library is in progress. This library will enable the user to reference files outside of the store. The imag-ref binary crate will be a frontend to use this library.

This is in progress and I'm really looking forward to it. I will use the libimagref library to write a wiki module with vimwiki as backend, after this gets merged.

store hooks: git

I want to continue to work on the git hook layer, so we finally have a version control system in the store.

Filter language

Another thing I really want to do is to implement the filter language frontend, so one can use libimagentryfilter to implement a filter right on the commandline.

The best option would be to add two/multiple frontends for this:

• hlua
• ketos (a lisp dialect)
• rhai

so one can select one of these languages by commandline and then filter entries.

That would be a really huge amount of work, but it would be fun, really.

LGPL checker

Another thing what I want to do is to add the LGPL header in each source file. The point is: I want to check this via the travis CI setup, so travis checks whether all files contain the LGPL header. That is possible, I did not yet figure out how to do it properly, though.

tags: #linux #open source #programming #rust #software #tools #imag

As imag gets more traction, I get asked more often “What is its purpose?” or “What does it do?” – here's an explanation!

After I posted my 8th iteration on “What's coming up in imag” on reddit I noticed a huge increase of the number of stars of the project. It went from less than 20 stars up to over 50 within one week. Amazing!

But I also got questions: What is this “imag” thing you are writing – what does it do? So here's an attempt to explain.

A CLI PIM suite?

The README file in the imag source repository states that imag is a personal information management suite for the commandline. Its goal is

Create a fast, reliable, forwards/backwards compatible commandline personal information management suite which covers all aspects of personal information management, consists of reusable parts and integrates well with known commandline tools.

Which is exactly what it is.

Yes, but...

imag is a CLI PIM suite but tries to implement as little functionality as possible – how does it do that? Well, we try to re-use existing tools. We do not implement the PIM tools ourselves, but we re-use known tools, so a user does not have to learn a new tool when switching to imag – she can simply add imag to her setup!

(Bad visualization skills ahead!)

As the picture above shows, all these fancy tools have some sort of database where they store their data. This is awesome and works – what it doesn't provide is to connect this data and add annotations to the connections – that's where imag steps in:

As the picture above shows, imag tracks the data other tools create (it does not duplicate it). It provides ways to link this data and therefor creating a network of data you can query later.

Ah, and what does work by now?

Not that much, as it is an really early stage right now. Some of the modules are already implemented, namingly the

• counter module
• notes module
• diary module

as well as some utility modules for more plumbing-like commands:

• imag-link for linking entries together
• imag-tag for adding, removing and altering tags of an entry
• imag-view for viewing entries. It only supports printing to commandline as of this writing, but will support more in the future, including showing dependency graphs of entries, compiling markdown and showing it in the browser and so on.
• imag-store A commandline-interface to the store itself. This shouldn't be used by the end-user and is for debugging purposes and development work. You can destroy data with this tool rather easily, so if you want to play around, make sure you have a backup of your data!

All tools mentioned above are implemented without using third-party software, which means that there are no tools other than imag which need to be installed to use the advertised functionality (of course imag itself depends on libraries). Future work will be based on other software, so the todo module will be based on taskwarrior, the ledger module will be based on beancount and so on.

The plan

Currently, I'm having in mind to add these tools to the imag suite:

• Todo module: taskwarrior
• Ledger module: beancount
• lend-tracker module: [no backend tool found yet] – I guess I will implement this myself as I cannot think of any complex functionality here.
• bibliography module: [no backend tool found yet] – I doubt that there are commandline tools available for organizing bibtex files. You are welcome to proof me wrong!
• bookmark module: [self-implementation] – I will implement bookmark functionality myself if I do not find a nice commandline tool which fits my needs.
• shopping list module: [not decided yet] – I want to create a module which helps me doing my shopping lists. I'm not sure whether there is a CLI tool for this available – I doubt it actually.
• calendar module: [self-implementation] – I will implement this functionality myself using the icalendar data format if I do not find some nice tool. Maybe khal is good enough, but I'm not sure it offers an API.
• contact module: [self-implementation] – Same as for the calendar module: I will implement this myself using the vcard data format if I cannot find a nice commandline tool. Maybe khard fits my needs, but I'm also pretty sure that this one does not offer an API either
• cue cards module: anki (not sure here, as anki is not a commandline tool or does not have a CLI interface (yet?))
• mail module: [self-implementation] – I'm pretty sure I have to implement this myself, because beeing able to reference emails means that I have to be able to read each of the mails anyways. Maybe this will be a complex module offering functionality to open mutt for mails/mailboxes and query sup or notmuch for more information about mails... - The possibilities are rather endless in this case and I think this will be the most complex tool in the imag suite.
• image module: [self-implementation] – The image module needs to be able to read the meta information of images and give me the possibility to query them for meta-information. I guess I can implement this myself.
• movie module: [self-implementation] – Here goes the same as for the image module – a module to query meta-information of movie files.
• music module: [self-implementation] – and another one equal to the image module – a module to query meta-information of music files.
• news module: newsbeuter – I'm a bit insecure because newsbeuter does not offer an API and uses a sqlite database as data storage. I would need to read this database and try to get my information from there. This is do-able in rust, but I'm not sure whether I like it or not. There should be a non-curses CLI interface for newsbeuter, really! This not only includes RSS and ATOM, but maybe also usenet and podcasts.
• password module: password-store – This is a home-run, as I won't do much besides creating pointers inside the store to each password file – so absolutely no complexity here!
• weather module: [not decided yet] – I'm not sure what commandline tools are out there that fit my needs here. For starters, I just want to be able to put weather information into the store and refer to it from other files. I've not read into the topic at all by now, so I won't promise things here.
• weight tracker module: [no backend tool found yet] – Are there commandline tools for weight tracking available? I would love to have one!
• workout module: [not decided yet] – This just came up while writing this blog post. Pretty much the same as for the weight tracker module... this could even include the weight tracker module!
• wiki module: [no backend tool found yet/not decided yet] – There are some wiki CLI tools available, but I'm not that happy with either of them. I would use vim-wiki personally, but it is rather hard to learn all the commands and I'm pretty sure I would screw up pretty bad when starting to use it actively. I wouldn't include vim-wiki in imag either, because I want to be able to be editor-agnostic.
• project management module: This is nothing more than a brain-fart by now, but maybe I can add a module to do issue tracking and project notes and so on. This shouldn't replace a bug tracker or something, just a personal-project-helper tool or something like that.

Infrastructure needed

I already wrote a lot of infrastructure libraries for imag, including but not limited to functionality for listing, viewing, tagging, linking filtering entries.

There are also libraries for user interaction, selecting entries on the commandline, markdown parsing or time specifying on the commandline to enhance the user experience.

Some more libraries for error handling, store hooks and runtime functionality are in the imag codebase for simplifying the process of writing imag modules.

And, of course, there is the very heart of imag, the store library. This is the library which ties everything together and the univeral interface to the filesystem for all the functionality in imag.

There is also a library in progress for referencing files which do not live in the imag store which can be used to “link” files into the imag store by creating pointers to them. It doesn't use symlinks, as symlinks can be killed by moving the original file. It also doesn't use hardlinks, as the user would have to remove two files instead of one when she removes data from her system – and this is clearly not feasible. So this library simply implements some kind of pointer to files. It also hashes the content of files to be able to re-find a file if it is moved (as long as the content does not change, of course).

The most important part which is almost not covered yet is testing. There is so much code to test.

My wishes for the future.

I have a varity of hopes and wishes for the future. The most important one is

Contributors!

I really really hope for contributors. I mean, I really hope some people start contributing. imag has grown to be my never-ending-project, as there are so many things to do, but I also think such a tool was not created before – so there is a lot of potential to discover and things to discuss or try out!

I think, though, I have layed out the groundwork on this. I created a list of issues in the github repository which are marked with the complexity/easy label so there's a list of easy-to-do issues which a contributor could check and try out.

I also write a two-weekly blog series on what is going on in the project (1, 2, 3, 4, 5, 6, 7, 8, 9) – I also post these on reddit in the /r/rust subreddit – so I really try to spread the word.

Flexible backends

This is another big thing. I wrote down a list of tools I want to integrate into imag above. All functionalities should re-use existing tools to cover there needs, I do not want to reimplement things others have implement very well before.

But, maybe someone does not like my choises and wants to use some other tool than I have used to integrate into imag. Or the maintainer of the external tool stops working on the tool – what then?

I want to create an abstraction layer over the backends, so one can simply switch from (for example) taskwarrior to imaginary-other-todo-tool.

That would be a really complex library, as it has to be completely tool and use-case agnostic. But I think it is do-able, just not yet, as we do not have the experience of binding several tools to imag.

Moving away from github (actually not...)

I'm also tempted to move away from github. The only thing that holds me there is the urge to find contributors and github is the easiest way to do so in my opinion. Also, if I move away from github, I really want to use text-only tools for developing imag – just as in the spirit of imag itself. The problem is, that there is no plain-text bug tracking software available that fits my needs (I wrote about that before). I could use a mailing list and ask contributors to send patches by mail. But I don't like that approach that much.

Also, some of the conveniences that github offers are rather nice. If I would do the move, I would setup a website, CI system, issue tracker, git repositories including a web-frontend (gitweb or similar, not a full-blown gogs or something), wiki, mailinglist, ... and import the old data from github, of course.

But as said, the tooling is not available for doing bug tracking in plain text and distributed, so I won't move.

To end this wall of text...

I didn't expect this article to get this long. But now it is. I hope I could explain what the purpose of imag is.

Feel free to ask me questions (via mail, you will find it) or suggest improvements to this article.

tags: #linux #open source #programming #rust #software #tools #imag