Really, how f*ck up is this travis thing?

In my current project imag, we use travis for CI. And in one of my branches, it fails because the bash scripts I use to test a binary cannot find the actual binary.

You know what the fix is? To call tree before running the scripts!

I mean, seriousely? How much can you propably mess up before things get this weird?

Fix your shit, travis people, really!

---

Update:

I mean... really? Now, the build succeeds on one test machine but fails on the other, because the build log gets too long for travis? You are freakin' kiddin' me, aren't you?

tags: #programming #software

I did it! Just hours ago I released imag in version 0.2.0.

Read here why I did that, despite there are few things working and I do not even consider this stable or even stableish.

First of all: This is not production-ready software. Use at your own risk! This is a release for people to notice that some things work and one could start to play around with it. Do not trust imag with your personal data if you do not have backups. There are bugs. This is not perfect. This is alpha quality or pre-alpha quality software!

imag is a personal information management suite for the commandline. Its target audience are commandline- and power-users. It does not reimplement personal information management (PIM) aspects, but re-uses existing tools and standards to be an addition to an existing workflow, so one does not have to learn a new tool before beeing productive again. Some simple PIM aspects are implemented as imag modules, though. It gives the user the power to connect data from different existing tools and add meta-information to these connections, so one can do data-mining on PIM data.

How the release was done

What did I do to get to the release? Well, not that much. I checked out the master branch and created a v0.2.0 branch on it, then started to fix things up for the release. That included:

• Moving the dependencies for all crates to “0.2.0”. That means that all crates in the imag code repository do not depend on other crates from the imag repository by their path but via crates.io and the “0.2.0” version of the crate. I had to do that to be able to publish these crates. This change wasn't done in master, so we can continue to work on multiple crates at the same time and do not have to cargo publish each crate independendently before beeing able to use it in other imag crates.
• I fixed the dependency markup in imag-store (actually two times) and the version information in libimagbookmark and imag-bookmark (in two seperate commits). Why is that? Well... because nobody is perfect. I simply missed them when updating the version strings from “0.1.0” (which is generated by cargo new) to “0.2.0”.

Then I manually published libimagutil and libimagerror because they do not depend on other imag crates themselves. After that I cargo published all crates in a loop until all of them were published. In the end I cargo published bin (the imag crate itself) and .imag-documentation (the documentation crate which does nothing by itself and just depends on all other crates for building the documentation).

I removed the v0.2.0 branch and created an annotated tag for the release.

Why now

I released imag 0.2.0 today because there are only few things left in the milestone for the 0.2.0 release and all whats left is tagged with release/optional, saying that these things are optional for the release (the issues will move to the next milestone, so they might not be there anymore when you click on this link).

The commit I based the release branch on was more-or-less random. It does not mark any special point in time or something, I just started to branch off of it.

Why releasing if not ready?

Why did I do the release if the software is not ready and nowhere from stable or even stableish?

That answer is pretty easy: To have a point to refer to and to get into a cycle (a release cycle). I'm pretty confident that this version is a point where interested people (read: interested developers and possible contributors) could start playing around with the software. Clearly, bugs are there and things will break. But, as imag is a rather complex piece of software and a large project, I wanted to have a point where people can be notified “Hey, this is working(ish) and you could try it out” so they can jump on the train.

And that's really what I want. I do not want to develop this monstrous project alone. I want the collaborate on this and hear what people want and need. This project solves my personal problems with PIM, but I'm sure that others will benefit from it as well and therefor community efforts are deeply encouraged by me. Or at least I hope that I do communicate this clear enough.

What's in there

Do not use imag if you do not have backups. See above for more detailed warning note.

What we have by now:

• imag-bookmark – A tool to keep and organize your bookmarks
• imag-counter – A simple counter tool. Create, delete, increment and decrement counters
• imag-diary – A diary
• imag-link – Link imag entries with other imag entries
• imag-notes – Create notes
• imag-ref – Reference files on your Filesystem with imag
• imag-store – Core plumbing tool. Do not use at all if you do not know what this is for.
• imag-tag – Tag imag entries
• imag-todo – Todo management with imag, using taskwarrior as backend. This is not a replacement for taskwarrior, but can be used to create entries in the imag store for each taskwarrior task and then these entries can be linked with imag-link.
• imag-view – View entries from the imag store
• imag – The imag binary itself. The purpose of this is to be able to call imag view instead of imag-view.

Clearly this is not everything. We have a huge number of libraries which work under the hood of these executables, but users might not care about this.

What's coming

Well, what's coming for the next release? The 0.3.0 milestone already exists and there are a lot of things already registered in it.

When will I be there? Well, my initial goal was to release 0.2.0 before 01-01-2017, so I clearly managed to get it out in time. I also announced that I want to do 3-month release cycles, so the next release would be due by March 31, 2017. I will keep that goal as I'm really not sure how fast I will be. Maybe we can release 0.3.0 early, but as normal in open source: The next version will be there if its ready.

I hope to land a imag-mail module in the next few weeks, so one can start tracking email with imag. I also hope to land imag-annotation to be able to annotate other entries with notes. imag mv will also be a thing that should go into 0.3.0 as well as more imag-view functionality.

And, of course, there will be a ton of enhancements to the libraries in imag.

Can I use imag libraries in my project?

No.

Well, you can, but I do not guarantee any stability or API consistency or something. But if you want, you clearly can use one of the imag libraries (they are on crates.io, so why not). Just keep in mind that interfaces will change and things will break and bugs will happen. We are 0.2.0 here, not 1.x.y. Keep that in mind.

For the curious, non-developers: imag consists not of one command/binary but of many. Each of those has a library accociated with it: libimagdiary and imag-diary where the imag-diary project is only a commandline interface to libimagdiary. There are some general purpose libraries in the imag codebase – for example libimagerror which are used by all other crates for a certain thing, error handling in this case. These libraries can be used independendently of imag itself. As said above, I do not guarantee anything in regard of API stability or such, so one could use them outside of imag, but I do not encourage it.

What now?

Well, I won't change my day-to-day rhythm when developing imag. As said, the 0.2.0 release was done to get some attention on imag and to people notice that some things might work, to play around and to report issues. 0.3.0 will be the same kind of release, but with more features. I don't know when I will be confident and say “Yep, one could now start to actually use this”, but I guess that it won't be 0.5.0 or 0.6.0 (which are not even planned by now). Maybe it will be 0.10.0 – who knows.

It also really depends on whether there will be (long term) contributors or not. If some people start working on imag more frequently and I can get a small community build up around the project, imag might be in a usable state “pretty soon” (read: within 2017 or 2018).

If not, ... I don't know. It's really that kind of project that runs forever. Does that bother me? A bit, ... but working on imag is fun and a pretty nice hobby... so I don't care that much.

That beeing said... feel free to play around by installing the imag crates (and having backups) and maybe you will be in the git log of the next release?

tags: #linux #open source #programming #rust #software #tools #imag

I wrote about imag before. I wrote about the problems I see with PIM on the commandline and how I want to solve them.

Well, it happens that some of my friends stepped up and reviewed the implementation (in manner of Software architecture) and suggested a design-overhaul. So this is what is happening right now.

As I stated before, I wanted the whole project to be a bit like git: each subcommand is an own executable, you can easily re-use parts of the code in your own projects as library and so on. This was one major point why we removed the whole codebase and started from scratch. Well not really from scratch, as the history is preserved and we can re-use parts of it by digging into the git-history. And, of course, the experience is not lost at all.

But before we did that, I set up a document where we can specify and document what we do. The major point of the paper is to define how the modules should work and what kind of interface they should export.

Some of these things are work in progress, some other things are already defined.

The roadmap changed a lot, of course. As we have now several libraries to implement and later glue together, we have a lot work to do on defining interfaces and (re-)implementing functionality.

I consider this as great opportunity to learn a whole bunch about software architecture.

We also applied some tweaks to the repository. We have now homu for merging and editorconfig so we are all in the same coding style.

The paper is written in pandoc-Markdown, so the paper can be read in either Markdown with your favourite text editor, PDF or as web-page (HTML). Both of the latter have to be compiled first, of course.

I hope enough people care about this project, so we can make it happen as fast as possible. I really want this to succeed, as I would personally benefit a lot from it.

More posts on this topic will follow, of course.

tags: #linux #open source #rust #software #tools #imag

In my blog article On the sad state of PIM for nerds I ranted over the lack of a PIM (Personal Information Management) suite for the terminal (and therefor for nerds like me).

I said that such a suite would be awesome to have and I listed some core features such a suite should have. I also said that I might not have the time for this. I was wrong. I just released version 0.0.1 of “imag” today.

“imag” stands for “Information ManaGement”. Not that creative, though I like this name. I just released version 0.0.1, which is in pre-alpha shape. I also only contains one “module” of the whole idea – a bookmarking module, where you can store your bookmarks and add tags to each of them.

The roadmap

There is a roadmap on github, this is the idea:

• Release v0.0.2 : Notes Module.
• Release v0.0.3 : BM + Notes combination, where one can link to bookmarks from a note
• Release v0.0.4 : Show meta-module for showing content of arbitrary modules
• Release v0.0.5 : Todo Module. Might wrap taskwarrior here. Not decided yet.
• Release v0.0.6 : Calendar Module, using icalendar standard
• Release v0.0.7 : Contacts Module, using vcard files

These are the releases planned so far, more planned Modules are (in alphabetical order):

• (Shopping)Lists Module
• Bibliography-Management Module
• Diary
• Ledger – to track personal finance
• Mail Module – to index email and link to from other content
• Movie library tracker – to index movies and link to from other content.
• Music library tracker – to index music and link to from other content.
• News Module – RSS/Atom, maybe using external tools like newsbeuter, undecided.
• Podcast module – implementation details completely undecided here
• Project planner – implementation details completely undecided here
• Wiki Module – expansion of the notes module with subpages and so on

Other things to do

There are more things planned in the github issues, like output formatter, where we can print all things with JSON (for example) so one can include “imag” in scripts. Also, an integrated webserver where one can view content as website would be nice (though I don't like the idea of editing content via a web page). Basically I'm open to all ideas.

Also one major paint point right now is the following: I have not yet integrated git support. So the file store is not yet version controlled, and I really have to do this soon, so I'm save.

I also did not think about encryption yet. I am thinking about encrypting the store and decrypting it on every call to “imag”, but I'm not sure whether this is the right way, as some application might call “imag” in background later (see next chapter). I'm not even sure whether I should encrypt the store at all, as the user should encrypt his or her device anyways.

Implementation, License, call for contributors

Well, as I want to be safe, I implement “imag” in Rust, my new favourite programming language. If you want to learn Rust, you are welcome to join me, as I'm also still learning things.

“imag” is licensed under the terms of LGPL v2. This is because “imag” might be used to be integrated into other software, maybe someone wants to write a curses frontend for “imag” at some point? Maybe vim plugins? Or something similar? I'd welcome it!

That said, I have to implement the core features first. Linking between content is one of my desired core features and will follow as soon as the notes module is implemented. As the readme file also says, there is a lot to implement, but not that much modules with actual user input. Only four modules: Notes, Shoppinglist, though I guess I will implement this one without user-input support as it only holds a list, then the wiki module and the Bibliography module, whereas I'm not sure whether this one has user input as well. So basically two modules have actual user generated input (speaking of text a user is supposed to edit with an text editor). All other modules contain structured data (JSON) the user can edit via the commandline.

All this is not carved in stone yet and I'm happy to argue about the points I just mentioned. Also I'm happy to argue about more modules and of course I'd love to see pull requests on github!

So feel free to contribute!

tags: #linux #open source #rust #software #tools #imag

Almost all issues for the 0.2.0 release of imag are done.

Here are some notes how I want to do releases before the 1.0.0 version of imag, which, of course, is really not there yet. But I had to think about a decent release strategy for the 0.x.y releases, so here the notes.

imag is a personal information management suite for the commandline. Its target audience are commandline- and power-users. It does not reimplement personal information management (PIM) aspects, but re-uses existing tools and standards to be an addition to an existing workflow, so one does not have to learn a new tool before beeing productive again. Some simple PIM aspects are implemented as imag modules, though. It gives the user the power to connect data from existing tools and add meta-information to these connections, so one can do data-mining on PIM data.

First, all the things I note here apply to releases before 1.x.y, so basically all 0.x.y releases.

I want to do 0.x.0 releases with a bunch of new features every 3 months or something like this. This time range is not fixed, and releases will happen if they are ready, but 3 month is a good first idea, in my opinion.

I won't declare a fixed set of features for every new version, but move around some features as I like, but I try to keep the number of changes decent. For example, the 0.2.0 release has 49 closed and 13 open issues/prs at the time of writing. As I sometimes do more PRs per issue and sometimes just one PR for one issue, or even file PRs without an existing issue in the first place, this number might vary, but I guess about 100 issues/PRs per release should be a maximum.

What I want to do, and what I already do: I mark issues as

• optional if they are optional for the release they are attached to
• soft-required if I would like to get them into the release but wouldn't mind moving them to the next release
• hard-required if they have to be in the release.

I also set a release date. hard-required issues will get my attention first, I try to implement them as soon as possible and get them ready. After that I focus on soft-required things and after that on optional things.

hard-required issues might postpone a release date. soft-required and optional issues do not. But if an issue has to move to the next release, because the release-date is near and the issues are not solved by then, they will automatically advance.

That means, if an issue is optional in 0.2.0, it will get soft-required if it has to move to 0.3.0 and hard-required if it has to move to 0.4.0.

This way, one can easily calculate when a feature will definitively be implemented in the imag codebase.

---

Another thing I try to do: Plan ahead a bit. At the time of writing, the milestones for 0.2.0, 0.3.0 and 0.4.0 exist. I won't create a 0.5.0 milestone before 0.2.0 is done, so I try to plan ahead about two releases, which equals about six months.

---

I will do y-releases (as in 0.x.y) if I encounter a really serious bug. But as we do not guarantee anything right now (see below), I doubt that will ever happen.

---

All the things I wrote above are ideas how I want to do it. I will see how things work out and we'll see whether this is doable for me or not. As I already said somewhere, I'm starting my masters degree this week and I don't know how the workload will be. I really hope I can get things implemented as planned, but I might have to move issues to future milestones if I notice that things are not doable for me in the planned time.

Also, as we are in the 0.x.y release phase, I won't give any guarantees on stability and interfaces an so on.

tags: #linux #open source #programming #rust #software #tools #imag

My 2 cents on 32c3 – a short recap.

It. Was. Awesome.

No, really. I had not that much opportunities to talk to strangers, though it was an awesome congress and I really look forward to 33c3!

I talked a bit to the rust guys (less than expected, but anyways, yeah), watched several talks, including “fnord Jahresrückblick” and the Perl talk. They were sooo awesome! I really recommend to watch them online!

Sadly, I couldn't watch the “Security Nightmares” talk, as my train left at 18:30, but I watched it online and it was one of the most awesome talks.

As always after c3, I was completely exhausted afterwards (3 hours less sleep per night). But it was worth it.

I hope I have the opportunity to attend next year as well.

tags: #ccc #life #media #network #social

I wish you all a merry christmas and a I hope to see you at the 32c3 in Hamburg! tags: #life #social

I recently started to learn a new programming language: Rust. And I really have to say: This is the kind of language I always wanted. Some parts are not as nice as I'd like them, though most of the language, the tooling, the compiler and everything else is just so damn cool.

But first, for the new readers of my blog:

Where I came from

I learned Java as first programming language when I entered 11th grade. I learned the stuff really fast and I started to learn Ruby right after. This was about the time of Ruby 1.9.2 and 1.9.3 – and it really was an awesome time.

Later I learned C, all the way down from “Oh my gosh, why am I getting this segfault – oh, I didn't allocate properly” to linux kernel patches. I really have to say it was fun and I enjoyed learning it, though I really don't want to implement a larger project in C. C is for systems programming deep down directly at the hardware.

On my way from 11th grade until today, I learned a fair amount of languages, some of them just really briefly, others more closely: Ruby, PHP, Bash, Java, Haskell, C, C++, Scheme/Racket, Nix, Python, Lua, SQL, HTML/CSS, SASS, HAML, Latex, JavaScript, vimscript and, of course, lately Rust.

Where I don't want to go to

I have three languages I want to write in future. Maybe four. Let me elaborate:

Haskell

Nice language, though for academic use only IMHO. I really don't want to write Haskell for production systems, if possible. I still can understand why people do this and even when people start new projects with Haskell as programming language, though I wouldn't do that myself.

PHP

Do I really have to explain this? I mean, PHP was great in the '90s and maybe early 2000s. But today, there are much better alternatives available.

HTML/CSS

Yes, you kind of have to write these languages in web development, but I'd opt for SASS and Haml. They are both nice preprocessor languages which are much simpler to use.

JavaScript

I didn't learn much about JavaScript by now, though I don't think it is a language I would want to use except for website animations.

Scheme/Racket

Nice languages! I really enjoy reading them, though I don't see any value in them for large production systems. Their tooling just looks to bad (from the outside) and while I'd love to see better tooling here, I don't really care about these languages.

Java

Nope.

Python

I see a lot of value in Python for small scripts, but not for large production software like web frameworks or similar. Yes, this was a hint in direction of Django!

Lua

Nice language for embedded stuff and so on, though I don't really care too much about Lua either.

SQL

The standard when it comes to database querying. Though I really don't want to write these things by hand and I really would love to have ORMs widely available (and yes, every major language has an ORM, I know that and I like it).

Latex

Can we please just have pandoc succeed and write our stuff in Markdown? I mean, yes, Latex kindof works, but when it doesn't it just sucks as hard as hell!

vimscript

Another thing I don't want to have to touch.

C++

There are better alternatives available... this is why I'm writing this post!

Where I want to go to

Yes, there are languages I like. It's really just three/four, but they clearly exist: Ruby, C, (Bash) and Rust. Each of them with another purpose. Bash is in parentheses here because I kindof have to use Bash, let me explain this first:

(Bash)

Bash is syntactically just disturbing. I mean, its syntax just sucks, right? But it is the default on all Unix systems and that won't change, so for scripts, I just adapt to this default and keep my things in Bash.

I know there are alternatives like fish, zsh and whatever. I cannot use them because of two reasons: If I change my shell, I want to change it everywhere, means I would need to convert all my scripts. This is a huge amount of work. And, I'd still have to write bash when contributing to nixpkgs, and not contributing to nixpkgs is just no option for me.

So, for the sake of uniformity, I just keep everything in bash.

Ruby

Ruby is awesome. I love its syntax, I love its tooling (though I don't have to mess that much with it).

I just don't want to implement complex things in it. I started to learn rails and I really like rails, despite it is a monster from my perspective. There are other web frameworks for Ruby available which are much less complicated. I speak partly of Sinatra, but also of Lotus which seems quite nice to me (again, from the outside).

Of course there are others.

C

C is,... well C. It is awesome for small bit-fiddeling work and OS kernels. It is great for microcontrollers. It is just as simple as possible a high-level-assembler language can be.

I really love writing C, though I also think one has to avoid it whenever possible.

Rust

Now I finally hit the chapter this whole blog post is about.

Rust. Is. Awesome.

I mean, really. It comes with the tooling of Ruby, cargo the package manager/build tool/test runner/everything just works out of the box and as good as one can imagine. Working with dependencies just works (if you get your types right). The language itself is really nice. I mean, I worked with for about one month by now and I really enjoy every problem I encounter in manner of a learning experience. I learn with every problem and I also learn to avoid problems really quickly.

Rust gives me power to my hands I couldn't even imagine when I was in 11th grade. I can write high level code like in Ruby, but I can also go into detail and fiddle around with bits and everything, without having to worry about References and pointers, data races, bla bla bla (take that, C++)! If the compiler tells me that I've done everything right and it compiles my code, I can be sure that the program won't eat my head while running. It might not do what I want, but it does what is written down without crashing in random places!

So, with Rust you get:

• If it compiles, it works (Haskell)
• It is fast ©
• You can build complex applications with type safety (C++ (though “type safety” is a stretchable term in C++))
• You can write high-level code (Ruby)
• Easy syntax (Ruby)

Summary

So, to summarize: Bash for system scripting, Ruby for more complex scripting tasks, C for embedded/system programming, Rust for everything else from systems programming to building space rockets.

If you haven't learned Rust yet, go and give it a try! You will be amazed!

tags: #programming #open source #rust

From time to time it can happen that nixos-rebuild fails due to a package build error.

Here is how to fix this.

So your nixos-rebuild tells you something like this:

(hashes removed for readability)

building Nix...
building the system configuration...
these derivations will be built:
  /nix/store/<hash>-libvterm-v2015-02-23-src.drv
  /nix/store/<hash>-neovim-libvterm-2015-02-23.drv
  /nix/store/<hash>-neovim-0.1.0.drv
  /nix/store/<hash>-neovim-0.1.0-configured.drv
  /nix/store/<hash>-system-path.drv
  /nix/store/<hash>-dbus-conf.drv
  /nix/store/<hash>-unit-dbus.service.drv
  /nix/store/<hash>-unit-polkit.service.drv
  /nix/store/<hash>-system-units.drv
  /nix/store/<hash>-etc.drv
  /nix/store/<hash>-nixos-system-yuu-16.03pre71289.7ae05ed.drv
building path(s) ‘/nix/store/<hash>-libvterm-v2015-02-23-src’
builder for ‘/nix/store/<hash>-libvterm-v2015-02-23-src.drv’ failed with exit code 1
cannot build derivation ‘/nix/store/<hash>-neovim-libvterm-2015-02-23.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/<hash>-neovim-0.1.0.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/<hash>-neovim-0.1.0-configured.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/<hash>-system-path.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/<hash>-dbus-conf.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/<hash>-unit-polkit.service.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/<hash>-unit-dbus.service.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/<hash>-system-units.drv’: 2 dependencies couldn't be built
cannot build derivation ‘/nix/store/<hash>-etc.drv’: 2 dependencies couldn't be built
cannot build derivation ‘/nix/store/<hash>-nixos-system-yuu-16.03pre71289.7ae05ed.drv’: 2 dependencies couldn't be built
error: build of ‘/nix/store/<hash>-nixos-system-yuu-16.03pre71289.7ae05ed.drv’ failed

And you don't know what to do? Well, there's a solution to that, and the solution is rather simple, actually.

Step 1: Find the broken package

If the break happens due to an error in your configuration.nix, you should fix this error. Then you don't need to go on reading here.

If the build fails due to an error in the nixpkgs, as above, you should continue reading.

So we have to find the broken package. From the backtrace nixos-rebuild shows us, we can find out that the break happens due to an error in the libvterm package. This is clearly an error in the nixpkgs and we can fix this error only with the nixpkgs available, so...

Step 2: Get nixpkgs

If you havn't already cloned the nixpkgs repository, you need to do this first. You also need to fetch the unstable channel commit, which is provided by the nixpkgs-channels repository:

git clone https://github.com/nixos/nixpkgs && cd nixpkgs
git remote add channels https://github.com/nixos/nixpkgs-channels

Step 3: Check out the commit of your system

Now you should have a commit which shows the current unstable channel. You can check this like so:

nixos-version # 16.03.git.ab0a7e9 (Emu)
git show nixos-unstable # commit ab0a7e9

If the commit is the same as the hash which is shown from calling nixos-version you have found the channel commit your system is build from. You can now check this commit out and create a new branch in the nixpkgs from it.

git checkout ab0a7e9
git checkout -b nixos-unstable-fixes

Step 4: Search for fixes.

Sometimes there are already fixes available in the master branch of nixpkgs, but are not in unstable yet. You can simply do something like this:

1. Find out where the broken package lives
2. Get the log from the current unstable channel commit to master
3. Apply these fixes in your -fixes branch
4. Rebuild the system with the local nixpkgs

If there are no fixes available, you have to fix the build yourself and I cannot help you with that, because the build problems are often very package-specific.

We already know that libvterm is the problem here.

grep -rnil libvterm pkgs/
# or 'ag -l libvterm pkgs', which is much faster

This gives us four possible locations:

pkgs/top-level/all-packages.nix
pkgs/applications/window-managers/vwm/default.nix
pkgs/applications/editors/neovim/default.nix
pkgs/development/libraries/libvterm/default.nix

We also know (from the backtrace) that it has something to do with neovim. So we check the changes of this one first:

git log --oneline nixos-unstable..master -- pkgs/applications/editors/neovim/default.nix
2617919 neovim: fix wrong ad30764d68a

Well, we found a fix. Let's apply it!

git checkout nixos-unstable-fixes # just to be sure
git cherry-pick 2617919

Now we can rebuild our system with this patch applied:

sudo nixos-rebuild switch -I nixpkgs=/home/user/nixpkgs/

... and our system is updated. Now you can try to update packages the same way using the diff from nixos-unstable to master and apply the appropriate patches onto your -fixes branch. After a channel update you can simply rebase the -fixes branch onto the new channel commit. If rebase fails due to conflicts (which shouldn't happen that often) you can just hard-reset your -fixes branch onto the unstable channel commit and re-apply patches as needed.

tags: #nix #nixos #software

This post is about how to setup vim (vim_configurable) on NixOS with reusable parts and the possibility to replace vim with neovim without rewriting the whole setup.

configuration.nix

What we have in the configuration.nix file is simple:

environment.systemPackages = let
    vimPackages = import ./some/directory/vimPackages.nix pkgs;
    # neovimPackages = import ./some/directory/neovimPackages.nix pkgs;
in
    vimPackages ++ with pkgs; [ some other packages ];

You can see that we simply import some nix expressions and pass them the pkgs of from our configuration.nix file. Everything else is done in this file and some sub-files.

vimPackages.nix

So here starts the fun. We define a function which gets the pkgs variable and the lib variable from the outside scope (we don't pass the lib variable, though nix is smart enough to figure out what we mean). We already know that we need to return an array of packages, so we can return vim and some things we use in our vim scripts, like so:

# ...
[ vim pkgs.ctags ]

vim is not prefixed with a pkgs. here because we define it in the file itself. But first, we need to know what we want to override. vim_customize allows us to compile vim without GUI support, for example (there are a lot more options). We also want to build our vimrc and we want to put plugins into our vim setup, because this is the NixOS way of life.

customization = {
    vimrcConfig = (import ./customization.nix { pkgs = pkgs; });
} // { name = "vim"; };

Here we import our customization (vimrc, plugins, etc) and ensure that the executable is named "vim". We can now use this customization to customize vim_configurable:

custom_vim = pkgs.vim_configurable.customize customization;

and afterwards we override this derivation and set variables, what features should be compiled into vim:

vim = lib.overrideDerivation custom_vim (o: {
    ftNixSupport = true;
    gui          = false;
    luaSupport   = true;
    # some more
});

Now we put these parts together and into a function, so the whole file looks like this:

{ pkgs, lib, ... }:

let
  customization = {
    vimrcConfig = (import ./vim/customization.nix { pkgs = pkgs; });
  } // { name = "vim"; };

  custom_vim = pkgs.vim_configurable.customize customization;

  vim = lib.overrideDerivation custom_vim (o: {
    aclSupport              = false;
    cscopeSupport           = true;
    darwinSupport           = false;
    fontsetSupport          = true;
    ftNixSupport            = true;
    gpmSupport              = true;
    gui                     = false;
    hangulinputSupport      = false;
    luaSupport              = true;
    multibyteSupport        = true;
    mzschemeSupport         = true;
    netbeansSupport         = false;
    nlsSupport              = false;
    perlSupport             = false;
    pythonSupport           = true;
    rubySupport             = true;
    sniffSupport            = false;
    tclSupport              = false;
    ximSupport              = false;
    xsmpSupport             = false;
    xsmp_interactSupport    = false;
  });

in [
  vim
  pkgs.python
  pkgs.ctags
]

But that's not all to it. The customization.nix is where the real fun begins.

customization.nix

So this is the file where we define which plugins we want to use. There are some plugins which are not yet included in the nixpkgs repository, so we have to build these packages ourselves to be able to install them. Luckily, the nixpkgs provide some helper functions to build vim plugin packages rather easily.

Also, we build our vimrc file here, which consists of another bunch of script files.

vimrc.nix

The vimrc itself is written in viml of course. If you have a lot of vimrc configuration, you can split this file into several smaller ones. For example, I have split these files into the following structure:

./
├── colorschemes
│   ├── mm_neverness_share.vim
│   └── razorlight.vim
├── filetypes.vim
├── folding.vim
├── general.vim
├── gui.vim
├── insertmode.vim
├── mappings.vim
├── menuconf.vim
├── plugins
│   ├── # ... some things
│   └── ycm.vim
├── pluginconfigurations.nix
├── scripts.vim
├── searchnreplace.vim
├── statusline.vim
└── textediting.vim

So what we do in our vimrc.nix file (which later gets included in our customization.nix file) is: We include all these .vim files and put them into one giant string, which is then our vimrc file:

{ stdenv, writeText }:

let
    generic     = builtins.readFile ./vimrc/general.vim;
    textediting = builtins.readFile ./vimrc/textediting.vim;
    # ... more here

    plug = import ./vimrc/pluginconfigurations.nix;
in

''
    ${generic}

    ${textediting}

    # ... more here

    ${plug}
''

You might noticed the vimrc/pluginconfigurations.nix file. This one is just another layer of abstraction here and is basically the same as the vimrc.nix file:

let
    # some files
    ycm = builtins.readFile ./plugins/ycm.vim;
in

''
    # some variables
    ${ycm}
''

plugins.nix

Another file we need is the plugins.nix file. As said, not all vim plugins are in nixpkgs (yet). So we have to build some vim plugins ourselves (don't get scared here. This does not mean you have to run compilejobs yourself, most plugins don't contain any binaries so rebuilding the system with vim plugins installed is normally a matter of seconds).

So this one is also rather simple in the end. In the example below you can see how I build the vim-trailing-whitespace plugin. The version might be outdated, though.

{ pkgs, fetchgit }:

let
  buildVimPlugin = pkgs.vimUtils.buildVimPluginFrom2Nix;
in {

  "vim-trailing-whitespace" = buildVimPlugin {
    name = "vim-trailing-whitespace";
    src = fetchgit {
      url = "https://github.com/bronson/vim-trailing-whitespace";
      rev = "d4ad27de051848e544e360482bdf076b154de0c1";
      sha256 = "594769a6f901407609b635a5041966456bfd91b13437169a4562857544e1dca3";
    };
    dependencies = [];
  };

  # more?
}

Finally... put the parts together.

Now we can write down our customization.nix file, which imports the plugins.nix and vimrc.nix files and puts them in the right context, so the nixpkgs infrastructure understands how we want to have our vim package build:

{ pkgs }:

let
  # this is the vimrc.nix from above
  vimrc   = pkgs.callPackage ./vimrc.nix {};

  # and the plugins.nix from above
  plugins = pkgs.callPackage ./plugins.nix {};
in
{
  customRC = vimrc;
  vam = {
    knownPlugins = pkgs.vimPlugins // plugins;

    pluginDictionaries = [
      # from pkgs.vimPlugins
      { name = "youcompleteme"; }

      # from our own plugin package set
      { name = "vim-trailing-whitespace"; }
    ];
  };
}

Building

After putting these things together you can rebuild your system. The vim binary might be compiled from source, though this isn't that much of a problem because there are only few vim updates, so rebuilding vim will happen every four weeks or something, not on every system rebuild.

nixos-rebuild switch

neovim

As said, you can now use the infrastructure we just wrote to compile the very same configuration into neovim, including your vimrc, plugins and everything.

{ pkgs, lib, ... }:

let
  nvim = pkgs.neovim.override {
    # don't alias neovim to vim, yet.
    vimAlias = false;

    configure = (import ./vim/customization.nix { pkgs = pkgs; });
  };

in [
  nvim
  pkgs.python
  pkgs.ctags
]

Just make sure your vimrc settings don't mess with neovim. Some things are not supported by neovim, so you have to put them into if-else conditions like so:

if has("nvim")
  " we don't need this in nvim
else
  set clipboard=exclude:.*
endif

And that's it. tags: #nix #nixos #vim #neovim #programming