De-Facto standard in modern software development for version control is git. Git is awesome, works really well and can be applied to a varity of workflows. With git, one can use E-Mail lists as decentralized communication.

What git lacks is a distributed issue tracker.

In the following I use “DIT” for “Distributed Issue Tracker/Tracking” or “Decentralized Issue Tracker/Tracking”

Problems with distributed issue tracking

Some basic things which you want to do with decentralized issue tracking is using plaintext as data format. Using anything else is just insane, if you think of git as a plain-text version control system (in “we version-control text, not binary data”).

Secondly, you want your issues be right in your git repository, as issues are always related to code in our world.

Sync

One big problem is of course the synchonization of the content. With git, you do synchronization on a pull-request basis, but you don't want that for a issue tracking system, do you?

One would have to approve issues before they are propagated to the community of a project, which is complicated. One could use the E-Mail workflow which was introduced with git, git send-email and git format-patch, though most users do not even use these functionalities of git.

Comments

Of course issues include discussions which contain commits of up to N people. An DIT must be able to not only synchronize them but also print them in a sane way.

Status

Most issue trackers allow things like tagging, setting a status for an issue, assign issues to contributors of the project and so on. How would one model these things?

Data format

Of course, one would implement such a system for doing issue tracking in plain text (besides the points noted in “Binary Data” below). But how would one organize the comments, tags, status updates, labels, assignees, etc. etc.

One would need to define an both easy and powerful plain text data format for this.

Where to keep the content

Do you want to keep the issue content in the source code repository of your project? Do you want to use another subdirectory inside your repository just for issue tracking? Or should one store this content within the .git directory somehow (like git-annex does)?

Or do you want to have a dedicated repository for issue tracking, increasing the complexity of referencing from one repository to another?

Binary data

Sometimes it makes sense to check-in binary data with issues, such as generated PDF files (imagine a tex project where the output looks not nice and you want to file a proof of concept for a new design of the document), images (think of screenshots) or other debugging data (core dumps, large log files, sqlite databases or things like that).

References

When writing a comment in an issue, you might want to reference other comments or code lines. An DIT must have an ability to handle references and to display them in a sane format.

Authentication

How to authenticate someone so she is allowed to add issues to the issue tracking system if you have a distributed system without a central component which does the authentication for you?

One could solve this issue by using the E-Mail patch workflow support which is baked into git, but then you again have the problem that someone has to approve issues. For a large system, this might make sense, for smaller systems it is just hassle you don't want to have. Also this requires knowledge about git send-email most users of git don't have these days.

Workflow support

Of course, a wide range of workflows has to be supported by the system. I don't even know about different workflows for issue tracking, so I cannot write a lot about problems which might come up with a distributed issue tracker.

Integration into existing systems

To establish such a system, integration with existing platforms is desired. Integrating github issues, redmine, gitlab, gogs and all the other (centralized) platforms out there.

And, almost more important, two-way-synchronization is desired. If one files an issue on github, the DIT must get this issue somehow and it must be further distributed into the issue trackers of the other systems, including comments, status updates, labels, etc. etc.

Wiki software

Some issue trackers include a wiki software. It is not required for issue tracking, but would be a nice-to-have for an issue tracker IMHO.

Interface(s)

Another problem is the interface such an issue tracker has to provide. Of course, some users want the commandline as interface, but others want a nice and fancy-looking web-gui, graphical interfaces or some might want to have a email-based interface.

The commandline would be the core of the system, of course. A DIT which is supposed to use git or integrate well with git must implement a commandline interface, no question.

Lesser complicated Problems

Lesser complicated problems include full-text search, tool integration (writing plugins for vim, emacs and other lesser cool editors and IDEs won't be a problem after a good DIT system exists), GUI or web interfaces, ...

Existing systems and their problems

Some approaches on this problem exist, though all of them are unmaintained and stalled.

• bugseverywhere is the most popular implementation for DIT. It is unmaintained since end of 2012 (version 1.1.1), last commit 2013. It claims to provide a varity of backends, including git, darcs and mercurial.
• Ditz
• gitissiues
• cil
• DITrack
• git-issues
• git-case
• and many more dead projects

---

More resources

There are some more resources on the problem, including a blog post on the “Current State of distributed issue tracking” (2010).

The wikipedia site on distributed issue tracking lists some software solutions, including gitit (wikipedia) and ikiwiki (wikipedia).

---

Brain-Dump of ideas

Of course, listing problems is nice though might not result in progress or even ideas. Here go some ideas I have.

First of all I though one should implement such a system from scratch. While this seems to be the most flexible solution in terms of implementation details, it might double efforts.

My current thoughts are that one should re-use the codebase of bugseverywhere (or one of the other solutions) learn from it and start rewriting and going on from a certain point of the solution to solve the remaining issues. Using ikiwiki as distributed issue tracker (as listed by wikipedia) could be a nice idea.

One crucial point for me is that this kind of software should not be implemented in, for example, Ruby. One should implement such a software in python (despite the pain), bash or a similiar wide-spread language. Of course, implementing it in a language which is compiled to a binary would be nice as well (Thinking of either Rust or C++ here, not necessarily Haskell). Scripting would be faster progress and if the system is well-defined one could rewrite parts of it in other languages later on, of course.

---

Just my two-cents.

tags: #git #software #tools

I started a project some time ago, where I develop a template for a wiki which is statically compiled into a html page. As the template is more mature now, I want to introduce you wiki.template and explain why I wrote it.

Once upon a time...

If you read this blog you know that I'm a member of the NixOS community. The NixOS community has, of course, a wiki. And the quality of this wiki is bad. No really, it is just one large page (the landing page) linking to sub-pages which are more or less maintained and contain only snippets of information. Sure – there's a lot of content in there, but (especially beginners) one has to search for it. You can't simply find it.

That's why I wanted to have a new wiki for the NixOS community. And because I like git, I wanted the content to be stored in a git repository.

So I came up with the idea to recreate the NixOS wiki and start it from scratch. Of course, the content of the original wiki has to be migrated to the new wiki then, but this has to be done carefully and by hand to ensure that things are ordered and easy to find afterwards.

The requirements

The requirements where rather simple, but nevertheless important to me. As already said, I wanted to have the content version controlled with git. The templates and markups for other data should be version controlled as well, if possible. This way, the content (or whole wiki) can be distributed with git, hosted on github or similar hosting platforms and so on.

I wanted to be able to customize the style of the wiki completely and I wanted to be able to write snippets of markup which can be reused, such as a warning-alert template where you can pass custom text and get a red alert box in your wiki page. Pretty normal stuff for a wiki.

Syntax highlighting, TOC-generating, all these things should be integrated or available through plugins (or there should be the possibility to write such plugins).

So I searched for wiki software with git backends...

The state of git-backend-driven wiki software

The state of wiki software with git backend is bad. No, it is even worse. There are actually two projects I had a look at and one which is written in Perl (and I didn't have a look on this one, because Perl).

gitit

First of all, there is gitit. Gitit is a wonderful piece of Haskell software. It works great and is fast. But it has its issues:

1. No Templates. Period.
2. Content is checked in, but neither templates nor style information
3. Haskell. So writing plugins – nah, not really.
4. Just few plugins available

So it was pretty clear that gitit is not an option for me.

gollum

Gollum was the other alternative. Developed by github, ruby thing, sounds well, right?

But. There's a big but.

1. No templates
2. No plugins
3. No custom styling or at least no documentation about it

It was clear to me: No option!

The solution

Well, the solution? Just write it yourself! And that's what I did. I started to work on a template for a wiki which is statically compiled with nanoc. Later I switched to jekyll.

Compiling the wiki to a static site has several advantages. First of all, hosting the site is cheap. A common web server can host thousands of pages without even noticing. Of course, you are also protected against hacking and all this stuff, as you do not have the problems dynamic websites have.

But these are common advantages of static compiled sites. An even bigger advantage is, that contributions to the wiki must pass a review-process of some kind. Of course, I will not grant push access to the repository to anyone and of course nobody gets access to merge pull requests, besides myself. I will protect myself against pushes to master as well, so each change to the wiki has to be reviewed. And that's what matters when it comes to quality: Changes must be reviewed.

Besides this big advantage, I do not have to care about updates (of the compilersoftware). I can do them locally and roll back if they fail. I do not have to keep track of spam bots, security updates, user account data, etc. etc. etc.

So here's the story how I wrote the template:

wiki.template

I started the project with nanoc, the static site compiler I love. The project is at version 4 these days and the compiler is really really good. It is flexible, scales well, fast and a joy to work with.

I just failed integrating foundation, the CSS framework I wanted to use. I played around with Jekyll a bit, but wasn't able to get Haml working. But at some point, foundation seemd to be more important than Haml, so I switched to Jekyll.

I also thought of the option to build the site with Jekyll directly from the github repository and build a github page with it, just because this way I can save money for hosting. I still think of that option, but as I use Jekyll with plugins I cannot build it on github directly. So I searched for tutorials on how to build it with travis and deploy my github page from travis. This is documented and a known as working.

Current state of wiki.template

To get my stuff working, I had to use some plugins. And these plugins basically work for me, but the code was either a mess or lacked features I wanted to have. So what did I do? Right, I forked them and send pull requests to the maintainers.

At this very point in time, I have three pending pull requests to these plugins, two of them just normal “cleanup codebase” PRs, one feature. I also opened issues for other features I do not necessarily need but would like to see implemented. I would implement them myself, but well... not enough time.

So what is the current state? Well, wiki.template is basically usable. I want to integrate some more features and it completely lacks of documentation on how to use it.

But these things will be done soon and I hope I can then create a fork of it to start my NixOS Wiki.

The NixOS Wiki

I already announced the NixOS Wiki fork (semi) officially on the NixOS mailinglist. Of course, as always in such communities, a discussion started on whether to do such a thing or not, which software to use and whatever. I don't care about this, I will simply do this and we will see whether it succeeds or not. Discussion all over the place helps nobody.

tags: #git #linux #nix #nixos #open source #programming #software #wiki

Firefox – The browser we all love (or hate?) and use almost every day (or never). But what the hell is this thing doing, actually? I just wondered why this thing eats the most resources on my device – while doing nothing.

So here it goes: I just synced my complete music, movie, library and image database. I do this using git-annex. If you don't know git-annex, you should really check it out! It's a tool for keeping things in git without actually checking it in. Sounds strange, but that's it! SO, git-annex is written in Haskell by Joeh Hess, the Debian guy you probably know. It is an awesome piece of software and does a great job for me. I sync several thousand of music pieces with it (something above 42,000, actually), several hundret GiB of movies, a collection of books (including audio-books) which is about 3,000 files and my private image database, which is about 7,000 files. And I do this without worrying that I might lose something. I know I always have at least 2 copies of the stuff, for music there are always 3 copies of everything. If I want to put something on my notebook (which, of course, does not have a 3TiB drive in it but only a 256 GiB SSD) I say “git annex, get XYZ for me” and it fetches it from somewhere – I don't have to check all my drives manually.

So, that's git-annex. git-annex does some heavyish stuff when keeping track of my data. It creates hashes of the data on and on. It does a lot of IO, of course. It propably does more things I don't even know of – I'm just a user. But what does this all have to do with firefox, you might ask? Well... here it goes.

So I was just syncing my music and the other stuff to two external drives. It synced about 2,5 TiB to each drive, so we're speaking of about 5 TiB data which was checksummed, synced via slow USB2.0, etc. It was fast, anyways. While syncing, I hacked a bit on one of my projects and thought about eggs with bacon, as I still had no breakfast this morning. And on my second screen, there was the login of my owncloud opened in Firefox. Nothing more, it was just the login screen (you might know that owncloud re-loads every now and then, which results in page rendering and all this stuff – but no, there was just the login). And you know what? Firefox used more resources than the heavy git-annex syncing processes together. What the hell does this thing? I mean... it just has to display this freaking web page. There is no javascript animation, no flash plugin (uuuh, I just said an ugly word) – I don't even have flash installed – and no re-rendering because of page reload or some shit like this. Just displaying. Why the hell does this thing need more resources than another process which actually does something?

So, well ... Maybe I'll never know. If you know why Firefox needs that much resources, you're welcome to tell me!

This rant ends here. tags: #git #haskell #software #tools #network

I started a fork of a diary application which is written in bash, because I thought there are some things missing and some features could be added – and the repository of the original author was really old.

I started to pretty it up and implement some 21st-century things so it becomes a really nice note-taking application.

So, initially the application was called “diary” and developed on sourceforge, by Jamie Briggs. Someone created a fork, named “diary-f”, which added some features.

The lastest change in the original repository is from February 2013, which is almost two years ago by now. The last change in the repository of the fork is from three years ago – even longer, yes!

So, the application per se is really good one. It is just a small shell script, but it works really good for me. Anyways, I missed some features, namingly:

• Multiple separate entries per day
• Everything should be kept in git, so I know when I edited the articles
• GPG encryption everywhere

I started implementing multi-file support directly in the “diary-f” repository, but then decided to create a fork, as it won't be compatible with the original software anymore. By now, three things are done in the repository: I refactored the codebase to fit my coding style (well, it is still bash, isn't it?), mutli-file support as said and git support. I want to implement gpg support next and maybe the ability to add attachements to entries. And after these things are implemented, I want to be able to search the entry repository for certain entries in a simple way, which could be difficult, given the fact that everything is encrypted in there.

---

If you're interested in the application, I'd welcome pull requests for almost everything. Contributing guidelines are:

• Document your work, how do things work?
• KISS
• DRY

That's basically it. I'm looking forward to your patches! tags: #bash #git #open source #programming #shell #tools

Today I will teach a classmate of mine a bit about git. I searched the web for tutorials, so I can refer to them, but hey, there are thousands out there.

So I came to the question: Which one should I link? They have things in common, some of them are better, some or not so good. But some of them have really different approaches on how they teach git. And that's my point: If you want to learn git, you have to search for a tutorial which fits for you and only you. It could absolutely be that you love the tutorial I used when I learned git, but it could also be that you hate it.

What I would encourage you to do is searching 3 to 5 different beginner tutorials and always reading the first two paragraphs or something. Then decide which one you want to use.

There is no “best” tutorial for learning git. There are only ones that fit your needs and other whichs don't. For example, for my classmate, I personally would search some more technical examples which also covers the “how does it work internally” part, at least at the surface. But maybe she would use a really basic tutorial which only tells her how to get shit done... who knows?

And with this statement, I'm coming to the next point: When it comes to git, and you want to collaborate with others (or you want to write a piece of software and share it with others afterwards, which enables them to collaborate with you), there is one thing: Know. Your. Shit.

You really should know how you do certain things right, so you don't break the overall workflow of the project you contribute to. That sounds more complicated than it actually is. You basically have to know how to fix history if it got messed up (in manner of a lot of “Fixup abc” commits), how to update your local copy and rebase your work onto the changes of the upstream repository, how to do pull requests right and how to write a clean history. Again, that sounds more complicated than it is.

If you know how to do the basic things right, you will have a less hard time using git in collaboration with others.

For example, I have another classmate. He uses git for ... dunno... maybe 4 weeks or so. He still did not get the difference between his local copy, his remote copy and the upstream repository. This leads him to confusion every single time he wants to contribute. And he fucks up the history pretty often, resulting in pull requests the maintainer of the main repository (me, actually), rejects because there is stuff in it which does not belong there. For example merge commits where he merged the upstream master. Or the pull requests is the hell of an outdated one and I cannot merge it because of merge conflicts.

He doesn't know his shit, and it slows down the workflow of the whole project sometimes (actually, just mine, but whenever I have to struggle around with his damaged pull requests, I cannot do the actual work with the other pull requests).

So, again: Know. Your. Shit.

And every good git tutorial should teach you these things. If it doesn't, it is not a good one. So here goes my next advice:

If you do not understand in a git tutorial after reading it several times, switch to another one and restart. Yes, restart. Start reading the other tutorial right from the beginning. Why? Because:

1. The tutorial might has stated important things you're skipping when you switch directly to the point you stopped in the old tutorial with.

2. Re-reading things you already know memorizes it.

It is like learning a song on the piano: The more often you do it, the more you know it. But if you always play it from the start to the end, you know the start really good (as you play this one more often if you restart after failing), and the end not that good. That's why some people start at the end when learning a song on the piano: The play the last line, then the 2nd last and the last, and so on. When they made it to the very beginning, the have to re-learn the beginning several times and afterwards, they can play the song just perfectly.

That's a really basic trick, but very effective. Sadly, this does not work when learning something from a tutorial, because a tutorial has to be done sequentially. But I guess you get the point.

So, let me make my points up:

1. Know. Your. Shit.

2. Learning with tutorials is good, but they have to fit your taste and need. If you do not understand certain things after re-reading them several times, switch to another tutorial

3. Extra point: Take your time. Rom was not build in one day, and neither you have to get from beginner level to expert level in git just in one day.

4. Another extra point: Tutorials are good, but sometimes it is better to actually talk to someone (I know... yuck!)

5. Finally: When working on a collaborating team after you learned the basics: Ask people about how they did things and about their tricks (short: go on learning. Every. Single. Day).

That's it. Cheer up, you can make it! tags: #git #tutorial

Have you ever had a file in your git repository which you would like to split, but you also would like to preserve the history of the file?

Here's a way how to do it.

The Problem

We want to move a (two-line) file a.txt inside a git repository to b.txt and c.txt, whereas b.txt contains the first lines of the original file and the c.txt file the second one. You want to preserve the history of these two lines, as a simple git mv a.txt b.txt && cp b.txt c.txt or something similar would cause at least one of the files (the c.txt one in this case) to be a completely new file to git, and therefor have a single commit as history – the creation of it.

I also tested some other ideas, for example a git mv a.txt b.txt && git checkout HEAD a.txt. It results in the same issue.

The solution...

... is a merge conflict! Yuck, you say? No, not at all! It's a really simple merge conflict!

First, you create a branch which causes the merge-conflict later on. Then, you do git mv a.txt b.txt && git commit. You moved the a.txt file now to b.txt, obviously! Now, go to your new branch. The file is still a.txt there. Do a git mv a.txt c.txt && git commit. Maybe you guess what comes next.

Create a merge-conflict by going back to your branch and merging the temporary created branch git merge --no-ff <conflict-branch>. A merge conflict will come up, as you moved a.txt to b.txt one time and to c.txt another time. Now, simply, take both files.

Of course there is also the possibility to delete code in the files inside the branches. Then, you “splitted” the a.txt file into two files but preserved the history.

Awesome!

tags: #git

git is a distributed version control system by Linus Torvalds, the father of the Linux operating system. I use it on an everyday basis when developing code or even when writing texts (papers for my university or similar stuff).

I prepared a workshop on git for my local Unix group, called “Unfug” (german for “mischief” or “horseplay”) which is a shortcut for “Unix friends and user group” in September. Last week, at the 5th of December, the first part of the talk was delivered by me, today (12th of Dec.) the second part will follow. I arranged this workshop in two parts, as there is too much stuff to cover for one session. The normal length of such a talk is one hour, and I used 90 mins for the first part, which is much enough (and of course the length of normal lessons at our university)!

The sources

This workshop is also public available (LaTeX source) at github, as I version-controlled it with git! You can download it and use it for your own stuff under terms of Creative Commons License. For detailed information have a look at the license file, which is shipped with the repository.

The idea of the workshop

The idea of the workshop is not to get the participants from beginner level to professionals. It gives them a basic idea how to work with git (basics as committing and branching) and mentions all other stuff (rebasing, workflows, integration in bash, vim, etc.) and is therefor just kind of an overview. The participants should be able to google around wanted stuff on their own afterwards, as they already heared what is possible with git and how it is called.

Your contribution is wanted!

Please note that the workshop is written in german as well as the commit messages are. I would love if you would submit an english version if you want to, but I don't know if latex supports compilation to english or to german. If you have an easy solution for this, please send me a pull request for your patches! Questions and suggestions are heavily welcome!

tags: #git #tutorial

I added some bash scripts to a repository I currently maintain for a project at my university. We wanted some scripts to generate git repository statistics and parse them to CSV, to be able to plot them with gnuplot lateron easily. The idea is that we can simply invoke make to build the whole thing with updated git statistics and everything (it is a tex document). Here are some of these scripts...

The full history, please, James!

First, let us generate a CSV which contains the full history, including (short) commit hashes, author names, dates and the header of the commit messages:

git log —date=local —pretty=format:“%h%x09%an%x09%ad%x09%s”

As simple as git is, as simple is this script. It's kind of cryptic, as we print x09, which is a tabulator, but it works perfectly! You can also change the --date part to iso to generate ISO dates!

Weekend work, or what?

With this neat piece of bash, we can check how much work was done at weekend days:

for i in Mon Tue Wed Thu Fri Sat Sun do echo -en “$i\t” echo $(git shortlog -n —format='%ad %s'| grep “$i ” | wc -l) done

Example output:

Mon 53 Tue 58 Wed 85 Thu 69 Fri 49 Sat 16 Sun 32

Neat, isn't it?

Do we work at night?

Well, the stuff about the weekdays is really neat, but we can also check if the commiters are morning grouches:

for i in seq -w 0 23 do echo -ne “$i\t” echo $(git shortlog -n —format='%ad %s' | grep “ $i:” | wc -l) done

Example output for this:

00 14 01 9 02 0 03 0 04 0 05 0 06 0 07 0 08 0 09 1 10 5 11 10 12 6 13 15 14 69 15 13 16 21 17 39 18 44 19 21 20 17 21 23 22 36 23 19

As you can see, we work the whole day, from 9 AM until 1 AM! But I think as the project goes on, we will also fill the gap between 2 AM and 8 AM! I hope you found this stuff interesting! I don't know much about gnuplot, the gnuplot part will be done by another project member (hopefully), but I guess this should be easily parseable by gnuplot!

tags: #git