musicmatzes blog

We cracked the 70-stars mark on github! Wow. We reached the 2,000-commits-mark. Wow. We also reached the 500 PR/Issue mark. Another wow.

But there happened a lot more things in the last 14 days in imag, the personal information management suite for the commandline. Read it up here.

The past

Wow. 14 days can be so damn short. We got a few things done in the last 14 days, but first, I want to thank all contributors:

  • mario-kr who did a semester project at his university with some fellow students. He submitted a pull request to implement libimagtodo, which is a todo module for imag, currently using task-hookrs as backend for talking to taskwarrior. This is really great!
  • mario-kr also submitted some more PRs:
    • A bugfix for the bin/imag binary
    • A feature for task-hookrs to import a task from taskwarrior from a single JSON object.
    • A lot of suggestions and comments in issues what is wrong with task-hookrs.
  • jsirois opened two PRs after reading my this-week-in-rust entries where I submitted two issues which could be done by people wanting to learn some Rust: 531 and 532 – absolutely awesome!
  • I also want to thank impo who submitted a PR to build the docs to a manpage. This reminds me that I really should work on the docs. Either way, a big “Thank you!” to you, impo!

Wow, so much contributions! Awesome, keep it up, guys! Lets take a look on the merged pull requests.

PRs merged

The following PRs were merged in the last 14 days:

  • 468 added libimagref - a library to create references to files on the filesystem. Something like pointers to external data. This can be used to reference files outside of the store, track them by content (SHA1), their permissions, their location and so on. This will also be used to implement a list of other tools, such as a wiki module, a movie-, music-, images-module and so on. There are a lot of possibilities here.
  • 481 implements a frontend for libimagref for the commandline, so a user can use libimagref directly to create filesystem references.
  • 487 added an implementation of Display for StoreId. You can now println!("{}", id) – awesome!
  • 489 refactored the internals of the store a bit, by moving a function from Store to StoreId. This shouldn't change the use of libimagstore at all.
  • 496 made the CoreLister-Type from libimagentrylist more generic. You should be able to pass not only closures, but also normal functions now.
  • 500 contained a bug-fix for 468
  • 503 fixed a flaw in the Display implementation for StoreId
  • 513 adds a configuration option which makes it possible to deny altering hooks per aspect in the store hook execution
  • 517 fixed a bug in libimaginteraction which caused the ask_bool() (and possibly other) function to loop endlessly
  • 518 removes a unneeded dependency from imag-view
  • 519 puts the out directory in the gitignore file
  • 521 adds the implementation for libimagbookmark – the bookmark feature of imag (the library part).
  • 525 updates the documentation. Before, the documentation was a technical document. I do not believe that this will bring us any advantages anymore. This PR rewrites the document to be a user documentation.
  • 528 updates all the crate versions from “0.1.0” to “0.2.0”, as we currently develop imag in this newer version.
  • 531 removes some unused code and therefor cleans up the warnings the compiler gives us about the unused code. This PR was later superceded by 536 and 537.
  • 532 adds a fold_result utility function for all iterators and therefor closes
  • 533 fixed a header-field naming error in libimagref.
  • 534 introduces a (hopefully) temporary fix for libimagentrylink. The check, whether a StoreId object points to an external link store object or not was buggy as it assumed the PathBuf inside the StoreId to be absolute to the Store root.
  • 536 superceded 531
  • 537 superceded 531
  • 538 refactored some code in the store to remove duplication.

PRs about to be merged

And here are some PRs which are work in progress and might be merged in the next 14 days. These were opened in the last 14 days and I hope to close them (merged or not merged) fast... some of them seem to be pointless (the ones on the store refactoring for testing are kind of duplicated as I tried different approaches) so they might get closed unmerged.

  • 486 implements a git store hook, which then puts the store into a git repository and tracks changes within the store. I'm not sure whether I will be able to merge this, as I still have problems getting the git2 crate building, as zlib is somehow missing, though I added it to my nixos setup... we will see...
  • 497 implements an iterator type for libimagref, so we can iterate over references in a nice way.
  • 498 uses 497 in imag-ref (481) to list references if the user requests this.
  • 514 adds an abstraction layer for the filesystem which should simplify store testing option which makes it possible to deny altering hooks per aspect in the store hook execution
  • 522 adds the implementation for imag-bookmark – the frontend for libimagbookmark, which is implemented in #521 - so this is basically the other half of the feature.
  • 529 is an attempt to pass only store-absolute StoreId-instances to the Entry objects inside the store. This PR might be superceded by 530
  • 530 tries to rewrite the Store internals to use another StoreId type (EntryStoreId) inside the Store implementation which is always absolute to the Store root and never relative to the filesystem root. 499, which is absolutely awesome.
  • 535 introduces a feature for imag-counter, which gives you the possibility to add units to your counters (when creating them). So you can attach a meaning to the counters and everybody will know that this is not the count of your daily pushups but rather the number of jellybaby you've eaten!

The Future

So what will happen next? I opened quite a bunch of issues in the last 14 days, most of them are rather simple things to do!

Issues opened/I want to get done

We reached the 60-open-issues line as well in the last 14 days. This is not necessarily a bad thing, as these are not all bugs but also feature requests and so on. Here is a list of issues which were opened in the last two weeks. I hope I can close some of them within a few days (some of them might be already closed at the time of writing).

I'm not including a description of each issue here... this would yield this post waaaay to long actually.

Other things

Well, I am in the last phase of my bachelors thesis right now... so I would say: Plenty of time to work on imag, right?

I hope to get the bookmark implementation finished and the listing support for the reference utility merged. Besides that (and after the reference foo is merged) I would love to start implementing a backend for the diary or bookmark module by either using jrnl.sh or buku. Not because I use them as commandline tools (I'm not saying I wouldn't) but because they seem to be simple to adapt. Maybe I'm wrong... but well... lets see.

tags: #linux #open source #programming #rust #software #tools #imag

I just released the second version of my task-hookrs crate, a crate for interfacing with taskwarrior!

This release is release 0.2.0 – so nothing too serious by now, as we still prefix with 0., but anyways a few nice things were merged and it seems to work pretty well!

These features were added:

  • Annotation support
  • Optional values are no longer included in the output
  • import_task() for importing a single task JSON object

Bugfixes and other things were also done:

  • Dead code removed
  • Documentation written and updated
  • Links in documentation fixed
  • Import errors for rust beta fixed

And the crate is already published on crates.io.

So go ahead and use it!

tags: #linux #open source #programming #rust #software #tools

imag has made little progress in the last 14 days, as I'm really focusing on my bachelors thesis and exams right now. Anyways, I got some neat things done. Read here what happened in the last 14 days.

The past

Yes, my thesis is my main focus right now. This does not mean that imag is stalled, indeed I got four merges done in the last 14 days, and some more are waiting.

Developers Certificate of Origin

I added a Developers Certificate of Origin in a pull request. This means, that all developers/contributors of imag agree to the terms listed in this certificate.

Within this pull request I also added some notes how to contribute to imag without having a github account. People are welcome to send me patches for imag via mail! They just have to acknowledge that I will integrate them by opening a PR on github myself. Also, they have to add a "Signed-off-by:" tag in their commit messages, I will not include patches send by mail without this line.

I think that shouldn't be an issue.

Result::map_err_into()

This was the second pull request. It was followed by two pull requests which moved the codebase to use the feature from this PR.

What it implements: An additional method for the Result type, so one can use the Result::map_err_into() method now to minify error handling code in a sane way.

Before this pull request, one had to write the following code over and over again, to wrap an error type into the module-specific error type:

some_operation_that_might_fail()
    .map_err(Box::new)
    .map_err(|e| ModuleError::new(ModuleErrorKind::FailingOperation, Some(e)))

This quickly became rather ugly. With the new feature applied, we can write this now in a really short and neat way. The following is equal to the code snippet above:

some_operation_that_might_fail()
    .map_err_into(ModuleErrorKind::FailingOperation)

With the map_err_into() method doing the heavy lifting of packing the error into a new Box and creating an error instance with the passed error kind.

As this was implemented in the libimagerror code generator macro, this is available for all crates which use the libimagerror macro generate_error_types automatically.

The PRs that followed that improvement removed a lot of boilerplate code:

4 files changed, 63 insertions(+), 113 deletions(-)

The Future

Lets see what's in the pipeline. Most of the following things are “I want to do this” things, and I won't be able to do them that soon, as the exams are coming up. I really hope I can get some progress, still.

References

The libimagref library is in progress. This library will enable the user to reference files outside of the store. The imag-ref binary crate will be a frontend to use this library.

This is in progress and I'm really looking forward to it. I will use the libimagref library to write a wiki module with vimwiki as backend, after this gets merged.

store hooks: git

I want to continue to work on the git hook layer, so we finally have a version control system in the store.

Filter language

Another thing I really want to do is to implement the filter language frontend, so one can use libimagentryfilter to implement a filter right on the commandline.

The best option would be to add two/multiple frontends for this:

so one can select one of these languages by commandline and then filter entries.

That would be a really huge amount of work, but it would be fun, really.

LGPL checker

Another thing what I want to do is to add the LGPL header in each source file. The point is: I want to check this via the travis CI setup, so travis checks whether all files contain the LGPL header. That is possible, I did not yet figure out how to do it properly, though.

tags: #linux #open source #programming #rust #software #tools #imag

As imag gets more traction, I get asked more often “What is its purpose?” or “What does it do?” – here's an explanation!

After I posted my 8th iteration on “What's coming up in imag” on reddit I noticed a huge increase of the number of stars of the project. It went from less than 20 stars up to over 50 within one week. Amazing!

But I also got questions: What is this “imag” thing you are writing – what does it do? So here's an attempt to explain.

A CLI PIM suite?

The README file in the imag source repository states that imag is a personal information management suite for the commandline. Its goal is

Create a fast, reliable, forwards/backwards compatible commandline personal information management suite which covers all aspects of personal information management, consists of reusable parts and integrates well with known commandline tools.

Which is exactly what it is.

Yes, but...

imag is a CLI PIM suite but tries to implement as little functionality as possible – how does it do that? Well, we try to re-use existing tools. We do not implement the PIM tools ourselves, but we re-use known tools, so a user does not have to learn a new tool when switching to imag – she can simply add imag to her setup!

(Bad visualization skills ahead!)

All my tools

As the picture above shows, all these fancy tools have some sort of database where they store their data. This is awesome and works – what it doesn't provide is to connect this data and add annotations to the connections – that's where imag steps in:

What imag does for you

As the picture above shows, imag tracks the data other tools create (it does not duplicate it). It provides ways to link this data and therefor creating a network of data you can query later.

Ah, and what does work by now?

Not that much, as it is an really early stage right now. Some of the modules are already implemented, namingly the

as well as some utility modules for more plumbing-like commands:

  • imag-link for linking entries together
  • imag-tag for adding, removing and altering tags of an entry
  • imag-view for viewing entries. It only supports printing to commandline as of this writing, but will support more in the future, including showing dependency graphs of entries, compiling markdown and showing it in the browser and so on.
  • imag-store A commandline-interface to the store itself. This shouldn't be used by the end-user and is for debugging purposes and development work. You can destroy data with this tool rather easily, so if you want to play around, make sure you have a backup of your data!

All tools mentioned above are implemented without using third-party software, which means that there are no tools other than imag which need to be installed to use the advertised functionality (of course imag itself depends on libraries). Future work will be based on other software, so the todo module will be based on taskwarrior, the ledger module will be based on beancount and so on.

The plan

Currently, I'm having in mind to add these tools to the imag suite:

  • Todo module: taskwarrior
  • Ledger module: beancount
  • lend-tracker module: [no backend tool found yet] – I guess I will implement this myself as I cannot think of any complex functionality here.
  • bibliography module: [no backend tool found yet] – I doubt that there are commandline tools available for organizing bibtex files. You are welcome to proof me wrong!
  • bookmark module: [self-implementation] – I will implement bookmark functionality myself if I do not find a nice commandline tool which fits my needs.
  • shopping list module: [not decided yet] – I want to create a module which helps me doing my shopping lists. I'm not sure whether there is a CLI tool for this available – I doubt it actually.
  • calendar module: [self-implementation] – I will implement this functionality myself using the icalendar data format if I do not find some nice tool. Maybe khal is good enough, but I'm not sure it offers an API.
  • contact module: [self-implementation] – Same as for the calendar module: I will implement this myself using the vcard data format if I cannot find a nice commandline tool. Maybe khard fits my needs, but I'm also pretty sure that this one does not offer an API either
  • cue cards module: anki (not sure here, as anki is not a commandline tool or does not have a CLI interface (yet?))
  • mail module: [self-implementation] – I'm pretty sure I have to implement this myself, because beeing able to reference emails means that I have to be able to read each of the mails anyways. Maybe this will be a complex module offering functionality to open mutt for mails/mailboxes and query sup or notmuch for more information about mails... - The possibilities are rather endless in this case and I think this will be the most complex tool in the imag suite.
  • image module: [self-implementation] – The image module needs to be able to read the meta information of images and give me the possibility to query them for meta-information. I guess I can implement this myself.
  • movie module: [self-implementation] – Here goes the same as for the image module – a module to query meta-information of movie files.
  • music module: [self-implementation] – and another one equal to the image module – a module to query meta-information of music files.
  • news module: newsbeuter – I'm a bit insecure because newsbeuter does not offer an API and uses a sqlite database as data storage. I would need to read this database and try to get my information from there. This is do-able in rust, but I'm not sure whether I like it or not. There should be a non-curses CLI interface for newsbeuter, really! This not only includes RSS and ATOM, but maybe also usenet and podcasts.
  • password module: password-store – This is a home-run, as I won't do much besides creating pointers inside the store to each password file – so absolutely no complexity here!
  • weather module: [not decided yet] – I'm not sure what commandline tools are out there that fit my needs here. For starters, I just want to be able to put weather information into the store and refer to it from other files. I've not read into the topic at all by now, so I won't promise things here.
  • weight tracker module: [no backend tool found yet] – Are there commandline tools for weight tracking available? I would love to have one!
  • workout module: [not decided yet] – This just came up while writing this blog post. Pretty much the same as for the weight tracker module... this could even include the weight tracker module!
  • wiki module: [no backend tool found yet/not decided yet] – There are some wiki CLI tools available, but I'm not that happy with either of them. I would use vim-wiki personally, but it is rather hard to learn all the commands and I'm pretty sure I would screw up pretty bad when starting to use it actively. I wouldn't include vim-wiki in imag either, because I want to be able to be editor-agnostic.
  • project management module: This is nothing more than a brain-fart by now, but maybe I can add a module to do issue tracking and project notes and so on. This shouldn't replace a bug tracker or something, just a personal-project-helper tool or something like that.

Infrastructure needed

I already wrote a lot of infrastructure libraries for imag, including but not limited to functionality for listing, viewing, tagging, linking filtering entries.

There are also libraries for user interaction, selecting entries on the commandline, markdown parsing or time specifying on the commandline to enhance the user experience.

Some more libraries for error handling, store hooks and runtime functionality are in the imag codebase for simplifying the process of writing imag modules.

And, of course, there is the very heart of imag, the store library. This is the library which ties everything together and the univeral interface to the filesystem for all the functionality in imag.

There is also a library in progress for referencing files which do not live in the imag store which can be used to “link” files into the imag store by creating pointers to them. It doesn't use symlinks, as symlinks can be killed by moving the original file. It also doesn't use hardlinks, as the user would have to remove two files instead of one when she removes data from her system – and this is clearly not feasible. So this library simply implements some kind of pointer to files. It also hashes the content of files to be able to re-find a file if it is moved (as long as the content does not change, of course).

The most important part which is almost not covered yet is testing. There is so much code to test.

My wishes for the future.

I have a varity of hopes and wishes for the future. The most important one is

Contributors!

I really really hope for contributors. I mean, I really hope some people start contributing. imag has grown to be my never-ending-project, as there are so many things to do, but I also think such a tool was not created before – so there is a lot of potential to discover and things to discuss or try out!

I think, though, I have layed out the groundwork on this. I created a list of issues in the github repository which are marked with the complexity/easy label so there's a list of easy-to-do issues which a contributor could check and try out.

I also write a two-weekly blog series on what is going on in the project (1, 2, 3, 4, 5, 6, 7, 8, 9) – I also post these on reddit in the /r/rust subreddit – so I really try to spread the word.

Flexible backends

This is another big thing. I wrote down a list of tools I want to integrate into imag above. All functionalities should re-use existing tools to cover there needs, I do not want to reimplement things others have implement very well before.

But, maybe someone does not like my choises and wants to use some other tool than I have used to integrate into imag. Or the maintainer of the external tool stops working on the tool – what then?

I want to create an abstraction layer over the backends, so one can simply switch from (for example) taskwarrior to imaginary-other-todo-tool.

That would be a really complex library, as it has to be completely tool and use-case agnostic. But I think it is do-able, just not yet, as we do not have the experience of binding several tools to imag.

Moving away from github (actually not...)

I'm also tempted to move away from github. The only thing that holds me there is the urge to find contributors and github is the easiest way to do so in my opinion. Also, if I move away from github, I really want to use text-only tools for developing imag – just as in the spirit of imag itself. The problem is, that there is no plain-text bug tracking software available that fits my needs (I wrote about that before). I could use a mailing list and ask contributors to send patches by mail. But I don't like that approach that much.

Also, some of the conveniences that github offers are rather nice. If I would do the move, I would setup a website, CI system, issue tracker, git repositories including a web-frontend (gitweb or similar, not a full-blown gogs or something), wiki, mailinglist, ... and import the old data from github, of course.

But as said, the tooling is not available for doing bug tracking in plain text and distributed, so I won't move.

To end this wall of text...

I didn't expect this article to get this long. But now it is. I hope I could explain what the purpose of imag is.

Feel free to ask me questions (via mail, you will find it) or suggest improvements to this article.

tags: #linux #open source #programming #rust #software #tools #imag

This is the 9th iteration on what's coming up in imag, the personal information management suite for the commandline.

First of all, I want to thank and welcome kbknapp who contributed a Makefile which enables to build the imag tools with just one command: make. Thanks a lot for that!

What happened

I wasn't able to do that much in the last 14 days, as my bachelors thesis wants some attention as well. But I was able to merge some nice PRs anyways, so here we go.

New issues!

In the last 14 days I tried to open as many issues as possible, tagging them and therefor making them accessible by possible new contributors (as some of them are tagged with the complexity/easy tag).

Diary!

Finally, the initial implementation of the diary module was merged! Yay! It already has some features I really like, but isn't feature-complete yet and there might be bugs. But the initial codebase is merged and I'm really happy about that.

One can use imag now as a personal diary. I did not integrate some diary tool into imag by now, because I thought implementing it myself would be kind of fun. Of course, I would like to add other tools as backend but I don't know of any (and I do not use any) I would find sufficient for integrating into imag.

More libraries and library-functionality!

Two awesome PRs were merged which added new functionality.

The first one is the “move” functionality for the store. It is not yet used by any other crate, but gives the possibility to save a copy of a store entry or move a store entry to a new location within the store. This functionality is not yet tested, but as we are pre-release anyways, this is not an issue in my opinion.

The other functionality is the libimagentrymarkdown which adds markdown-parsing functionality. There is just the initial codebase for this crate added by now, but the library will make it possible to parse the contents of a store entry to a web page and view it in the browser, for example.

The other merges

I merged some rather critical bug fixes, for example one were an entry wasn't edited correctly when calling edit_content() (from libimagrt) on it. A code-cleanup PR for libimagentrylink and some other fixes were merged as well.

The future

So lets see what's in the pipeline at the moment.

I'm working on a new crate called libimagref which gives access to primitives and functionality to reference files which do not live inside the store. For example to reference some music files, the libimagref can be used to generate a reference in the store which tracks the file(path), optionally having the possibility to check the metadata of a file by generating a hashsum over its contents and/or permission settings.

Some tests for the tagging binary (imag-tag) are work-in-progress.

For the next 14 days I hope to finish the libimagref implementation as well as getting some progress in the libimagentryfilter/lang PR, which aims for a DSL implementation in the entry filter crate, giving the user access to a filter language to filter out entries on the commandline like so (diary example here):

imag diary list --filter 'e.size() > 5.kilobytes() && e.lines().first().empty()'

Of course, the upper example is not how it will look like, as I'm still not sure what DSL implementation I will use and how I will model the access to the entry data, but the idea will be the same.

Another thing I really want to test out is quickcheck. I guess this could be really useful to test imag library crates properly and I hope to get something up in the next 14 days in this regard.

tags: #linux #open source #programming #rust #software #tools #imag

There is this game, slither.io. It is a really neat amusement (which btw also heats your linux device if you do not have a decent graphics driver) – and here's the trick how to be good at it.

So, this browser game is a bit like a multi-player snake. You start in a world where you are a small snake and there is food around and you can grow by eating. You can use your mouse to steer your snake. If you click the mouse button, your snake speeds up (you have to press and hold the mouse button). That costs you length, but if you do it well and you eat more than you lose length while speeding, everything is fine. You cannot die by this, as at some point, you simply stop speeding as you have no power left and you have to eat more to continue speeding. With your head you can cross your own body, so that's different from normal snake. What you cannot do – crossing other snakes. If you try to do that, your snake dies and turns into foo. Food gets also counted in points and the game shows a ranking who is #1 at the moment.

That's all to it.

So how can you grow fast? You could either try to eat a lot or (and here it comes) you trick other players (snakes) into running into your body by speeding before their head. The other snake then turns into food and you can eat it. Of course, longer snakes turn into more food than shorter ones.

If a snake dies and is at a certain size, all snakes (players) nearby try to get the biggest part of the cake by speeding to the new apprearing food. This often results in crashes – even more food!

The point is: How greedy are you? If you see a snake die and turn into food - you surely want to eat all the food. But you can only do that by speeding through the amount of food – increasing the likelihood of crashing into another snake and therefor turning into food yourself.

So here's the point how you can be really good at this game (I, myself, played it only about ten times before I got the longest snake with ~27k points) and I was able to hold this place for 10-15 minutes. How did I do that? By not beeing greedy – really, that's all to it! If you see another snake die, there's no problem with speeding to it and getting a huge amount of food out of it. But if you try to get as much as you can, you'll propably kill your snake.

tags: #social

Another 14 days are over. Here's my update what's coming up in imag, the personal information management suite for the commandline.

The past

We had 40 closed issues and pull-requests in the last 14 days, which is really awesome.

I was able to merge 29 pull requests in the last 14 days, which is quite a lot, so I won't elaborate on all of them (some of them are just small fixes), but let me do a short iteration on the highlights:

  • We implemented Store::get() which returns a None if the entry does not exist (Store::retrieve() implictely creates the entry).
  • The Store::get() functionality was also implemented in the imag store, imag notes and imag tag commands, causing them to fail if you try to list or view non-existent store content.
  • We have a new Hook which gets executed by the store after the store is loaded (so it is called only once, but hooks might do some fancy stuff before the user commands are executed).
  • We have a new crate for writing Date and Time interfaces, which helps parsing user-specified date and time. This crate is not tested yet, though some tests exists within the crate and they succeed.
  • Another crate was introduced for querying the user to provide a store id if he hasn't already, providing a fuzzy-search interface based on the interactor crate. This crate is not tested yet.
  • The error crate libimagerror provides a way to define custom members and custom code for the error types now, beeing more flexible now.
  • libimagerror also implements Into for the error kinds now, making the code using it even less complex.
  • All crates were rewritten to use the infrastructure provided by libimagerror
  • The binary at bin/imag was rewritten from Bash to Rust.
  • Updates for some dependencies were done
  • Bug fixes, including but not limited to
    • A bug in the config-finding algorithm in libimagrt
    • Bugs in the Hook execution in libimagstore
    • Duplication of data in libimagstore (FileLockEntry and Entry had the StoreId stored, despite the one dereferences to the other)

I also had (finally) some progress with my diary implementation. I guess I may finish it within the next 14 days. I'm serious this time, as the current code works and almost all the things are implemented at the time of writing.

I also have some (little) progress with the bookmark implementation, though I'm not sure whether I will implement this (rather) simple tool by myself or re-use existing solutions. On the one hand the problems are rather simple to solve with the existing imag infradstructure, on the other hand, I don't want to double work. I guess I will end up implementing it myself anyways, as I consider the (external-)link code part of the infrastructure and I don't want to depend on an external tool which might be or get unmaintained at some point in future.

I also have a new crate in my pipeline, which helps with Markdown parsing. So for the first step, we will provide Markdown only. But I will wrap this crate libimagentrymarkdown into a more generic one (libimagentrymarkup) to be able to write more backends for it (thinking of textile, asciidoc and so on, maybe by using a pandoc interface) and support other markup languages later on without hassle.

All in all, I really made some progress with the project in the last 14 days. I wrote more than 80 commits in the four days of the GPN last weekend, which was a huge amount of the work in the last two weeks, of course. I hope I can continue my streak.

The future

Well, the future. I don't really think I can keep up with my latest streak, though I really hope getting the diary implementation done. Bookmarks would be nice as well.

One thing I'm thinking about almost constantly in the back of my head is the DSL implementation for libimagentryfilter. I really want to dig in a little deeper into this. Maybe I can get something working within the next 14 days and if it proves useful I might extend the DSL so I can script the Store with it. Of course, this won't be a replacement for Rust as implementation language, but might prove useful for smaller scripts and the like? I'm not sure.

I also hope to find some contributors for the project. If you are looking for rust projects to contribute to, feel free to ask me questions or simply try to implement one of the complexity/easy tagged issues on github. Of course you can also ask questions there, I'm happy to answer them!

tags: #linux #open source #programming #rust #software #tools #imag

De-Facto standard in modern software development for version control is git. Git is awesome, works really well and can be applied to a varity of workflows. With git, one can use E-Mail lists as decentralized communication.

What git lacks is a distributed issue tracker.

In the following I use “DIT” for “Distributed Issue Tracker/Tracking” or “Decentralized Issue Tracker/Tracking”

Problems with distributed issue tracking

Some basic things which you want to do with decentralized issue tracking is using plaintext as data format. Using anything else is just insane, if you think of git as a plain-text version control system (in “we version-control text, not binary data”).

Secondly, you want your issues be right in your git repository, as issues are always related to code in our world.

Sync

One big problem is of course the synchonization of the content. With git, you do synchronization on a pull-request basis, but you don't want that for a issue tracking system, do you?

One would have to approve issues before they are propagated to the community of a project, which is complicated. One could use the E-Mail workflow which was introduced with git, git send-email and git format-patch, though most users do not even use these functionalities of git.

Comments

Of course issues include discussions which contain commits of up to N people. An DIT must be able to not only synchronize them but also print them in a sane way.

Status

Most issue trackers allow things like tagging, setting a status for an issue, assign issues to contributors of the project and so on. How would one model these things?

Data format

Of course, one would implement such a system for doing issue tracking in plain text (besides the points noted in “Binary Data” below). But how would one organize the comments, tags, status updates, labels, assignees, etc. etc.

One would need to define an both easy and powerful plain text data format for this.

Where to keep the content

Do you want to keep the issue content in the source code repository of your project? Do you want to use another subdirectory inside your repository just for issue tracking? Or should one store this content within the .git directory somehow (like git-annex does)?

Or do you want to have a dedicated repository for issue tracking, increasing the complexity of referencing from one repository to another?

Binary data

Sometimes it makes sense to check-in binary data with issues, such as generated PDF files (imagine a tex project where the output looks not nice and you want to file a proof of concept for a new design of the document), images (think of screenshots) or other debugging data (core dumps, large log files, sqlite databases or things like that).

References

When writing a comment in an issue, you might want to reference other comments or code lines. An DIT must have an ability to handle references and to display them in a sane format.

Authentication

How to authenticate someone so she is allowed to add issues to the issue tracking system if you have a distributed system without a central component which does the authentication for you?

One could solve this issue by using the E-Mail patch workflow support which is baked into git, but then you again have the problem that someone has to approve issues. For a large system, this might make sense, for smaller systems it is just hassle you don't want to have. Also this requires knowledge about git send-email most users of git don't have these days.

Workflow support

Of course, a wide range of workflows has to be supported by the system. I don't even know about different workflows for issue tracking, so I cannot write a lot about problems which might come up with a distributed issue tracker.

Integration into existing systems

To establish such a system, integration with existing platforms is desired. Integrating github issues, redmine, gitlab, gogs and all the other (centralized) platforms out there.

And, almost more important, two-way-synchronization is desired. If one files an issue on github, the DIT must get this issue somehow and it must be further distributed into the issue trackers of the other systems, including comments, status updates, labels, etc. etc.

Wiki software

Some issue trackers include a wiki software. It is not required for issue tracking, but would be a nice-to-have for an issue tracker IMHO.

Interface(s)

Another problem is the interface such an issue tracker has to provide. Of course, some users want the commandline as interface, but others want a nice and fancy-looking web-gui, graphical interfaces or some might want to have a email-based interface.

The commandline would be the core of the system, of course. A DIT which is supposed to use git or integrate well with git must implement a commandline interface, no question.

Lesser complicated Problems

Lesser complicated problems include full-text search, tool integration (writing plugins for vim, emacs and other lesser cool editors and IDEs won't be a problem after a good DIT system exists), GUI or web interfaces, ...

Existing systems and their problems

Some approaches on this problem exist, though all of them are unmaintained and stalled.

  • bugseverywhere is the most popular implementation for DIT. It is unmaintained since end of 2012 (version 1.1.1), last commit 2013. It claims to provide a varity of backends, including git, darcs and mercurial.
  • Ditz
  • gitissiues
  • cil
  • DITrack
  • git-issues
  • git-case
  • and many more dead projects

More resources

There are some more resources on the problem, including a blog post on the “Current State of distributed issue tracking” (2010).

The wikipedia site on distributed issue tracking lists some software solutions, including gitit (wikipedia) and ikiwiki (wikipedia).

Brain-Dump of ideas

Of course, listing problems is nice though might not result in progress or even ideas. Here go some ideas I have.

First of all I though one should implement such a system from scratch. While this seems to be the most flexible solution in terms of implementation details, it might double efforts.

My current thoughts are that one should re-use the codebase of bugseverywhere (or one of the other solutions) learn from it and start rewriting and going on from a certain point of the solution to solve the remaining issues. Using ikiwiki as distributed issue tracker (as listed by wikipedia) could be a nice idea.

One crucial point for me is that this kind of software should not be implemented in, for example, Ruby. One should implement such a software in python (despite the pain), bash or a similiar wide-spread language. Of course, implementing it in a language which is compiled to a binary would be nice as well (Thinking of either Rust or C++ here, not necessarily Haskell). Scripting would be faster progress and if the system is well-defined one could rewrite parts of it in other languages later on, of course.

Just my two-cents.

tags: #git #software #tools

I'm at the GPN16 at this very moment and I was amazed by one event which tool place while beeing outside and hacking on some Python code.

A guy came to me and told me that he just took a nice picture of me. He showed me the picture, which was indeed really nice, and asked whether this is okay or whether he should delete it.

I love that! He asked for permission, which is really awesome. Of course he did after he took the photo, but that's okay with me, as he took it with a camera and not an Android or Apple device or something like this, where I would assume instant sync to the cloud or something like that.

I really like beeing asked about this. I told him that I don't like beeing photographed and he deleted it instantly and also explained what he was doing (“Look here, Delete –> 'Delete this photo?' –> 'Yes' –> Photo deleted”).

Really awesome!

tags: #life #social #privacy

Is it more important to learn things or to have progress? And especially, is it more important to have progress in an open source project or is it more important that people learn new things?

I got a pull request for my open source project imag which wasn't well-formatted. As we are all nice people in the rust community (not only because, but also because we have this code of conduct which basically says you should be nice to all people) I asked the contributor to clean up the history by rewriting it.

Basically, I had two issues with the PR:

  1. The blank lines at the end of the files were removed. Some editors do this by default, but I don't like it as it makes diffs which add content at the end of a file rather complex (visually). If you have the blank at the end of the file, the diff is just one block which adds content, normally.
  2. I asked to rewrite the commit messages but after this was done, only one big commit was left. This is not preferred style in my project, I'd like to have more small commits than big monolithic ones.

So I asked kindly whether the contributor cared to fix up the issues. He did! Awesome! But he also told me that he was new to git and didn't knew how to do it. So I told him the steps which needed to be done, of course. He got stuck at some point (yes, git can be challanging, I know that) and asked for more help while apologizing for the slow progress.

And that made me start thinking about whether it is more important to have progress or that people learn new things. The issue which was adressed by the pull request described above is a rather simple one. It is literally a no-brainer. For me. But maybe not for others. Maybe also for others, but the review process isn't and the “Please rewrite the history like this” request was the challange. Either way, there was a point where the contributor had to learn something new which caused the pull request to slow down.

Is it worth it?

I honestly think yes. Learning new things is way more important than getting progress in this kind of project. I myself had to learn a whole bunch of things before this project was on the way it is right now. I had to learn that in Rust, you have to add more types to cover complexity and the more types you have, the better is your setup and the more extensible it gets. Of course this is not always the case, but I would say mostely.

I had to learn a lot of things. I had to learn to fail and to retry, which are two really important things. I failed so hard, I had to restart my whole project (which had no contributors by then, so no problem).

At the time of writing, the pull request is open for 8 days. It touches about 100 lines of code by adding missing calls. It could be done by myself within 3 minutes. But in my opinion, it is way more important that we all learn something and have fun. I offered to do it myself to the contributor, but clearly stated that if he wants to learn new things, he is free of taking his time and trying out how to fix the pull request. I really hope he learns something and starts contributing more often because of this.

tags: #software #tools #programming #open source