This post was written during my trip through Iceland and published much latern than it was written.

What is a nice and gold API. How is “nice” defined when it comes to library interfaces? That's a question I want to discuss in this post and also, how you can create a nice API in your open source library without studying a topic like software architecture or similar.

Definition of a “nice”/” easy to use” API

But first, we have to define what makes an API good. And that's not that easy because this topic is very biased.

For me, a good API is one where I can get the job done without thinking much about it. That means that there shouldn't be that much setup code involved in my code just to use the library. So no Factory hell if the only thing I want to have is the current time, for example. This also means that the API has to be decent high level, but without losing the ability to do fine-grained work if necessary. So for the most part, low level (for example implementation details) things are not interesting for me. But when I want to bit-fiddle around with the library, it should let me.

If a builder, factory or some other mechanism is necessary to produce objects in some way, the library should make clear (documentation wise but also code wise) why it is needed. There's no point in making the user call the tenth factory instantiation if it is not necessary and also it makes the users codebase blow up in size and complexity.

The naming of things in the library should be good, appropriate and, for the most part, be consistent. If a function on an object which returns the string representation of that objectbis named “to_string” it should be named that way for all types from that library, not only some parts.

Statelessness

Calling functions of your API should always result in the same values for the same arguments. That does not mean that your API should be pure in a functional programming meaning, but rather that the actions executed when calling a function should not result in some library-internal variables to be set, changed or unset. This is easily achievable by letting the user of the API have an object that holds the state, and functions of your API work based on that value. In short: your library should not have global variables.

This simple design pattern already results in easy to use APIs and a nice user experience.

Error exposure

Good libraries don't hide errors. Indeed, it is even better if errors are exposed to the user as much as possible. Because the user of the library knows best when and how to handle errors, even from your library.

I'm also a big fan of lots of error cases. The more error cases (the better a user of a library can distinguish between different errors) the better. This way, you let the user decide where she doesn't distinguish between two almost-equal error cases and where it is better to handle them independently. If your library does not give that opportunity, the user has to make ugly Spaghetti-code handling things to be able to tell what is going on. Of course, these things have to be documented properly.

Another thing that can come in handy is when your error types or your library exposes functionality to translate error types into text which can be shown to a user of your library. Nothing is worse (from a users point of view) than a “CallOnInconsistenStateObjectBuilderFactory on line 2832” error message shown in an user-facing interface (and trust me, I've seen such things already).

Completeness

Nothing is worse than an API that is not complete. I mean, don't get me wrong – sometimes one does not think of all cases a library could be used for – and that's completely okay. But some things are too obvious for being left out. For example, if you provide functions to transform your time object from local time into GMT, why wouldn't you provide functions for converting it into UTC or EST? These also matter!

Also cleanup routines. In some languages it is necessary to include cleanup routines for your objects. If your library exposes alloc_vacation_location_obj() it should also provide free_vacation_location_obj()! Sure, a user could use free(), but it is not nice API-wise. Even if your function does nothing more than call to free(), it is better to provide a function (and if you want to include some more cleanup in your function later on, in a new version of your library, a user does not have to think about it that much when upgrading their dependencies).

Consistency

We had the naming game already, but it always comes back to us, right? Consistent naming is one of the most important things in an API. If allocating worked with functions prefixed with new_ all the time, it shouldn't be done with alloc_ this time. Also not in later versions if your library. Not even in a major version bump.

Even more important than naming is behaviour. A function that is named with some alloc prefix should only allocate, never initialize or do other fancy stuff (debugging output excluded here, if necessary).

Next

In the next episode we will talk about how one can plan an application.

tags: #open-source #programming #software #tools #rust

This post was written during my trip through Iceland and published much latern than it was written.

When writing my last entry, I argued that we need decentralized issue tracking.

Here's why.

Why these things must be decentralized

Issue tracking, bug tracking and of course pull request tracking (which could all be named “issue tracking”, btw) must, in my opinion, be decentralized. That's not only because I think offline-first should be the way to go, even today in our always-connected and always-on(line) world. It's also because of redundancy and failure safety. Linus Torvalds himself once said:

I don't do backups, I publish and let the world mirror it

(paraphrased)

And that's true. We should not need to do backups, we should share all data, in my opinion. Even absolutely personal data. Yes, you read that right, me, the Facebook/Google/Twitter/Whatsapp/whatever free living guy tells you that all data need to be shared. But I also tell you: Never ever unencrypted! And I do not mean transport encryption, I mean real encryption. Unbreakable is not possible, but at least theoretically-unbreakable for at least 250 years should do. If the data is shared in a decentralized way, like IPFS or Filecoin try to do, we (almost) can be sure that if our hard drive failes, we don't lose the data. And of course, you can still do backups.

Now let's get back to topic. If we decentralize issue tracking, we can make sure that issues are around somewhere. If github closes down tomorrow, thousands, if not millions, open source projects lose their issues. And that's not only current issues, but also history of issue tracking, which means also data how a bug was closed, how a bug should be closed, what features are implemented why or how and these things. If your self-hosted solution loses data, like gitlab did not long ago on their gitlab.com hosted instance, data is gone forever. If we decentralize these things, more instances have to fail to bring the the whole project down.

There's actually a law of some sort about these things, named Amdahl's law: The more instances a distributed system has, the more likely it is that one instance is dead right now, but at the same time, the less likely it is that the whole system is dead. And this is not linear likelihood, it is exponential. That means that with 10, 15 or 20 instances you can be sure that your stuff is alive somewhere if your instance fails.

Now think of projects with many contributors. Maybe not as big as the kernel, which has an exceptionally big community. Think of communities like the neovim one. The tmux project. The GNU Hurd (have you Hurd about that one?) or such projects. If in one of these projects 30 or 40 developers are actively maintaining things, their repositories will never die. And if the repository holds the issue history as well, we get backup safety there for free. How awesome is that?

I guess I made my point.

tags: #open-source #programming #software #tools #git #github

This post was written during my trip through Iceland and published much latern than it was written.

Almost all toolchains out there do a CI-first approach. There are clearly benefits for that, but maybe that's not what one wants with their self-hosted solution for their OSS Projects?

Here I try to summarize some thoughts.

CI first

CI first is convenient, of course. Take github and Travis as an example. If a PR fails to build, one has not even to review it. If a change makes some tests fail, either the test suite has to be adapted or the code has to be fixed to get the test working again. But as long as things fail, a maintainer does not necessarily have to have a look.

The disadvantage is, of course, that resources need to be there to compile and test the code all the time. But that's not that much of an issue, because hardware and energy is cheap and developer time is not.

Review first

Review keeps the number of compile- and test-runs low as only code gets tested which is basically agreed upon. Though, it increases the effort a maintainer has to invest into the project.

Review first is basically cheap. If you think of an OSS hobby project, that might be a good idea, especially if your limited funding keeps you from renting or buying good hardware where running hundreds or even thousands of compile jobs per month can be done at decent speed.

What I'd do

I'm thinking of this subject in the context of moving away from github with one of my projects (imag, of course. Because of my limited resources (the website and repository are hosted on Uberspace, which is perfect for that), I cannot run a lot of CI jobs. I don't even known whether running CI jobs on this host is allowed at all. If not, I'd probably rent a server somewhere and if that is the case, I'd do CI-first and integrate that into the Uberspace-hosted site. That way I'd even be able to run more things I would like to run on a server for my personal needs. But if CI is allowed on Uberspace (I really have to ask them), I'd probably go for Review-first and invest the money I save into my Uberspace account.

tags: #open-source #programming #software #tools #git #github

This post was written during my trip through Iceland and published much latern than it was written.

From my last article on whether to move away from github with imag , you saw that I'm heavily thinking about this subject. In this post I want to summarize what I think we need for a completely self-hosted open source programming toolchain.

Issue tracking

Do we actually need this? For example, does the kernel do these things? Bug tracking – yes. But Issue tracking as in planning what to implement next? Maybe the companies that contribute to the kernel, internally, but not the kernel community as a whole (AFAIK).

Of course the kernel is a bad example in this case, because of its size, the size of the community around it and all these things. And other smaller projects use issue tracking for planning, for example the nixos community (which is still fairly big) or the Archlinux community (though I'm not sure whether they are doing these things only over mailinglists or via a dedicated forum at archlinux.org.

Bug tracking

Bug tracking should be done on a per-bug basis. I think this is a very particular problem that can be easily solved with a mailing list. As soon as a bug is found, it is posted to the mailing list and discussion and patches are added to the list thread until the issue is solved.

Pull request tracking

With github, a contributor automatically has an web-accessible repository. But for the most part it is sufficient if the patches are send via an email-patch workflow, which is how many great projects work. Having web-accessible repositories available is just a convenience github introduced and now everybody expects.

I think pull requests (or rather patchsets) are tracked no matter how they are submitted. If you open a PR on github, patches are equally good tracked as with mailing lists. Indeed I even think that mailing lists are way better for tracking and discussion, as one can start a discussion on each individual patch. That's not really possible with github. Also, the tree-shape one can get into when discussing a patch is a major point where mailing lists are way better than github.

CI

Continuous Integration is a thing where solutions like gitlab or github shine. They easily integrate with repositories, are for free and result in better and tested code (normally). I do not know of any bigger open source project that does not use some form of CI. Even the kernel is tested (though not by the kernel community directly but rather companies like Intel or Redhat, as far as I know).

A CI solution, though, is rather simple to implement (but I'm sure it is not easy to get it right). Read my expectations below.

How I would like it

Issue and bug tracking should be based on plain text, which means that one should be able to integrate a mailing list into the bug tracking system. Fortunately, there is such an effort named git-dit but it is not usable yet. Well, it is useable, but has neither email integration nor a web interface (for viewing). This is, of course, unfortunate. Also, there is no way to import existing issues from (for example) github. And that's important, of course.

For pull request/patch management, there's patchworks. I've never worked with it, but as far as I can see it works nicely and could be used. But I would prefer to use git-dit for this, too.

I would love to have an CI tool that works on a git-push-based model. For example you install a post-receive hook in your repository on your server, and as soon as there is a new push, the hook verifies some things and then starts to build the project from a script, which preferably lives in the repository itself. One step further, the tool would create a RAM-disk, clone the just pushed branch into it (so we have a fresh clone) and builds things there. Even one step further, the tool would create a new container (think of systemd-nspawn) and trigger the build there. That would ensure that the build does not depend on some global system state.

This, of course, has also some security implications. That's why I would only build branches where the newest (the latest) commit is signed with a certain GPG key. It's an really easy thing to do it and because of GPG and git itself, one can be sure that only certain people can trigger a build (which is only execution of a shell script, so you see that this has some implications). Another idea would be to rely on gitolite, which has ssh authentication. This would be even easier, as no validation would be necessary on our side.

The results of the build should be mailed to the author/commiter of the build commit.

And yes, now that I wrote these things down I see that we have such an tool already: drone.

That's it!

That's actually it. We don't need more than that for a toolchain for developing open source software with self hosted solutions. Decentralized issue/PR tracking, a decent CI toolchain, git and here we go.

tags: #open-source #programming #software #tools #git #github

This post was published on both my personal website and imag-pim.org.

I'm thinking of closing contributions to imag since about two months. Here I want to explain why I think about this step and why I am tending into the direction of a “yes, let's do that”.

github is awesome

First of all: github is awesome. It gives you absolutely great tools to build a repository and finally also an open source community around your codebase. It works flawlessly, I did never experience any issues with pull request tracking, issue tracking, milestone management, merging, external tool integration (in my case and in the case of imag only Travis CI) or any other tool github offers. It just works which is awesome.

But. There's always a but. Github has issues as well. From time to time there are outages, I wrote about them before. Yet, I came to the conclusion that github does really really well for the time being. So the outages at github are not the point why I am thinking of moving imag away from github.

Current state of imag

It is the state of imag. Don't get me wrong, imag is awesome and gets better every day. Either way, it is still not in a state where I would use it in production. And I'm developing it for almost two years now. That's a really long time frame for an open source project that is, in majority, only developed by one person. Sure, there are a few hundred commits from other, but right now (the last time I checked the numbers) more than 95% of the commits and the code were written by me.

Imag really should get into a state where I would use it myself before making it accessible (contribution wise) to the public, in my opinion. Developing it more “closed” seems like a good idea for me to get it into shape, therefore.

Closing down

What do I mean by “closing development”, though? I do not intend to make imag closed source or hiding the code from the public, that's for sure. What I mean by closing development is that I would move development off of github and do it only on my own site imag-pim.org. The code will be openly accessible via the cgit web interface, still. Even contributions will be possible, via patch mails or, if a contributor wants to, via a git repository on the site. Just the entry gets a bit harder, which – I like to believe – keeps away casual contributors and only attracts long-term contributors.

The disadvantages

Of course I'm losing the power of the open source community at github. Is this a good thing or a bad thing? I honestly don't know. On the one hand it would lessen the burden on my shoulders with community management (which is fairly not much right now), issue management and pull request management. On the other hand I would lose tools like travis-ci and others, which work flawlessly and are a real improvement for the development process.

The conclusion

I don't really have one. If there would be a way to include Travis into a self-hosted repository as well as some possibility for issue management (git-dit isn't ready in this regard, yet, because one cannot extract issues from github just yet), I would switch immediately. But it isn't. And that's keeping me away from moving off of github (vendor lock in at its finest, right?).

I guess I will experiment with a dedicated issue repository with git-dit and check how the cgit web interface works with it, and if it seems to be good enough I will test how it can be integrated (manually) with emails and a mailing list. If things work out smoothly enough, I will go down this road.

What I don't want to do is to integrate the issue repository in the code repository. I will have a dedicated repository for issues only, I guess. On the other hand, that makes things complicated with pull request handling, because one cannot comment on PRs or close issues with PRs. That's really unfortunate, IMO. Maybe imag will become the first project which heavily uses git-dit. Importing the existing issues from github would be really nice for that, indeed. Maybe I'll find a way to script the import functionality. As I want a complete move, I do not have to keep the issue tracking mechanisms (git-dit and github issues) in sync, so at least I do not have this problem (which is a hard one on its own).

tags: #open-source #programming #software #tools #git #github

This article is a cry for a feature which is long overdue in KDE, in my humble opinion: Syncable, user readable (as in plain-text) configuration files.

But let me start explaining where I come from.

I started with Linux when I was 17 years old. At the time I ran an Kubuntu 9.04 (if I remember correctly) with KDE 3. I disliked Gnome because it looked silly to me (no offense, Gnome or Gnome people). So it was simply aesthetics which made me use KDE. Before switching to Linux I had only experienced the cruel world of Microsoft Windows, XP at the time. When I got a new Notebook for my Birthday, I got Vista and two days later I had this Linux thing installed (which friends of mine kept talking about). Naturally, I was blown away by it.

After some time I got rather comfortable using this black box with the green text on it – the terminal finally had me! Soon, I launched a full blown KDE 3 (themed hackerlike in dark colors) to start a few terminals, open vim and hack away. Then, the same friend who suggested Linux told me about “tiling window managers”. wmii it was shortly after.

A long journey began. After some troubles with Ubuntu 12.04 I switched to Archlinux, later from wmii to dwm and after that to i3 which I kept for a few years.

In 2015 I learned about NixOS, switched to it at the beginning of 2016 and in late 2016 I reevaluated my desktop and settled with XFCE.

And here's the thing: I wanted KDE, but it lacked one missing critical feature: Beeing able to sync its configuration between my machines.

I own a desktop machine with three 1080p Monitors and two Notebooks – a Thinkpad X220 and an old Toshiba Satellite (barely used), if that matters at all.

There are things in the KDE configuration files which shouldn't be there, mainly state information and temporary values, making the whole thing a pain in the ass if you want to git add your configuration files to git push them to your repository somewhere in the web and git pull it on another machine.

Apparently the story is not that much better with XFCE, but at least some configuration files (like keyboard shortcuts, window decoration configuration and menu-bar configuration) can be git added and synced between machines. And it works for me, even with two so different machines. Some things have to be re-done on each machine, but the effort is pretty low.

But with KDE (I tried Plasma 5), it was pure PITA. Not a single configuration file was untouched, reordering values, putting state information in there and so on and so forth.

And I am rather amazed with KDE and really want to play with it, because I think (and that's not just an attempt to kiss up to you KDE guys) that KDE is the future of Linux on the desktop! Maybe it is not for the Hacker kid from next door, but for the normal Linux User, like my Mom, your Sister or the old non-techy guy from next door, KDE is simple enough to understand and powerful enough to get stuff done efficiently without getting in your way too much.

So here's my request:

Please, KDE Project, make your configuration files human readable, editor-friendly and syncable (via tools like git). That means that there are no temporary values and no state information in the configuration files. That means that configuration files do not get shuffled when a user alters a configuration via a GUI tool. That means that the format of the configuration files do not get automatically changed.

The benefits are clear, and there are no drawbacks for KDE as a project (as far as I can see) because parsing and processing the configuration files will be done either way. Maybe it even reduces the complexity of some things in your codebase?

---

A word on “why don't you submit this to the KDE project as a request”: I am not involved with KDE at all. I don't even know where the documentation, bug tracker, mailinglist(s), ... are located. I don't know where to file these things, whether I have to register at some forum or whatever.

If someone could point me to a place where nice people will discuss these things with me, feel free to send me an email and guide me to this place!

tags: #open-source #software #tools #git #desktop #linux

In this blog post, which might turn into a short series, I want to plan a rust library crate and write notes down how to implement it.

This article was yet another one that I wrote while being on my trip through Iceland. As you can see – my head does never stop thinking about problems.

Usecase

So first of all, I want to write down the use case of this library. I had this idea when thinking about how to design a user frontend for imag (of course) and came to the conclusion that rust lacks a library for such a thing. So why not writing one?

I want to design the user interface of this library crate approximately like Rails did with their implementation of the same functionality for Ruby (bear with me, I'm not that involved in the Ruby world anymore so I don't know whether this is actually Rails or just another gem that comes with it).

So what I want to be able to do is something like this:

let event_date = today() – days(2) + weeks(10);

for example. I'm not yet entirely sure whether it is possible to actually do this without returning Result<_, _> instead of real types (and because I'm in Iceland without internet connection, I cannot check). If results need to be returned, I would design the API in a way so that these functions and calls only create a AST-like object tree which then can be called with a function to calculate the final result:

let eventdate = today() – days(2) + weeks(10); let eventdate = try!(event_date.calc());

But even more ideas come to mind when thinking about functionality this library may provide:

// Creating iterators today().repeat_every(days(4)) // –> Endless iterator

// Convenience functions (today() + weeks(8)).endofmonth() // The end of the month of the day in 8 weeks

today().endofyear().day_name() // name of the day at end of the current year

today().until(weeks(4)) // Range of time from now until in 4 weeks

// more ...

Later on, a convenient parser could be put in front of this, so a user can actually provide strings which are then parsed and be calculated.

calculate(“now – 4h + 1day”)

Which then could of course be used to face a user as well.

Core Data types

As the foundation of this library would be the awesome “chrono” crate, we do not have to reimplement all the time-related things. This eases everything quite a lot and also ensures that I do not double work which others have done way better than I could have.

So at the core of the library, we need to encapsulate chrono types. But there are many user-facing types in chrono and we cannot assume we know which of them our users need. So we have to be generic over these types, too. This is where the fun starts.

At the very base level we have three kinds of types: Amounts (like seconds, minutes, etc, fixed points in time as well as time ranges:

pub enum TimeType { Seconds(usize), Minutes(usize), //... Years(usize), Point©, Range(A, B) } // A, B and C being chrono types which are wrapped

As I assume right now, we cannot simply subtract and add our types (and thus chronos types) without possible errors, so we have to handle them and return them to the user. Hence, we will create intermediate types which represent what is about to be calculated, so we can add and subtract (etc) them without error:

enum OpArg { TT(TimeType), Add(AddOp), Sub(SubOp) }

pub struct AddOp(OpArg, OpArg); pub struct SubOp(OpArg, OpArg);

trait CalculateableTime { calc(self) –> Result; }

with the trait implemented on the former types – also the enum maybe as I explain in a few words.

To explain why the CalculateableTime::calc() function returns a TimeType rather than a chrono::NaiveDateTime for example, consider this:

(minutes(15) – seconds(12)).calc()

and now you can see that this actually needs the function to return our own type instead of some chrono type here.

The OpArg type needs to be introduced to be able to build a tree of operations. In the calc implementation for the types, we can then recursively call the function itself to calculate what has to be calculated. As the trait is implemented on TimeType itself, which just returns Self then, we automatically have the abort-condition for the recursive call. To note: This is not tail-recursive.

Optimize the types

After handing this article over to two friends for some review, I got told that the data structures can be minified into one data structure. So no traits required, no private data structures, just one enum and all functions implemented directly on it:

enum TimeType { Seconds(usize), Minutes(usize), //... Years(usize), Point©, Range(A, B),

Subtraction(TimeType, TimeType), Addition(TimeType, TimeType) }

and as you can see, also almost no generics.

After thinking a bit more about this enum, I concluded that even things like EndOfWeek, EndOfMonth and such have to go into it. Overall, we do not want a single calculation when writing down the code, only lining up of types where the calculate function takes care of actually doing the work.

Helper functions

In the former I used some functions like seconds() or minutes() – these are just helper functions for hiding more complex type signatures and can hopefully be inlined by the compiler:

pub fn seconds(s: usize) –> TimeType { TimeType::Seconds(s) }

So there is not really much to say for these.

Special Functions, Ranges, Iterators

To get the end of the year of a date, we must hold the current date already, so these functions need to be added to the TimeType type. Ranges can also be done this way:

now().until(tomorrow()) // –> TimeType::Range(_, _)

Well, now the real fun begins. When having a TimeType object, one should be able to construct an Iterator from it.

The iterator needs to be able to hold the value it should increase itself every time as well as a copy of the base value. With this, one could think of an iterator that holds a TimeType object and every time the next() function is called, adds something to it and returns a copy of it.

Another way of implenting this would be to know how many times the iterator has been called, multiply this with the increase value and add this to the base.

I like the latter version more, as it does not increase the calculations needed for getting the real value out of the TimeType instance every time the iterator is called.

This way, one can write the following code:

let v : Vec<_> = now() .every(days(7)) .map(TimeType::calculate) .take(5);

to retrieve five objects, starting from today, each separated by one week.

Next

What I think I'll do in the next iteration on this series is summarize how I want to develop this little crate. I guess test driven is the way to go here, after defining the type described above.

---

Please note: This article was written a long time ago. In the meantime, I learned from a nice redditor that there is chrono::Duration which is partly what I need here. So I will base my work (despite beeing already started into the direction I outlined in this article) on the chrono::Duration types and develop the API I have in mind with the functionality provided by chrono.

For the sake, I did not alter this article after learning of chrono::Duration, so my thoughts are lined up like I originally had them.

tags: #open-source #programming #software #tools #rust

Here I want to describe how I plan to refactor the logging back end implementation for imag.

This post was published on imag-pim.org as well as on my personal blog.

What we have

Right now, the logging implementation is ridiculously simple. What we do is: On every call to one of the logging macros, the log crate gives us an object with a few informations (line number, file, log message,...) – we apply our format, some color and write it to stderr.

This is of course rather simple and not really flexible.

What we want to have

I want to rewrite the logging backend to give the user more power about the logging. As we only have to rewrite the backend, and the log crate handles everything else, the actual logging looks non different and “client” code does not change.

+----------------------------+
| imag code, libs, bins, ... |
+----------------------------+
              |
              | calls
              |
              v
+----------------------------+
| crate: "log"               |
+----------------------------+
              |
              | calls
              |
              v
+----------------------------+
| imag logging backend impl. |
+----------------------------+

So what features do we want? First of all, the imag user must be able to configure the logging. Not only with the configuration file but also via environment variables and of course command line parameters. The former will be overridden by the latter, respectively. This gives the user nice control, as she can configure imag to log to stderr with only warnings being logged but when calling a script of imag commands or calling imag directly from the command line, these settings can be temporarily (for the script or one command) be overridden.

The configuration options I have in mind are best described by an example:

# The logging section of the configuration
[logging]

# the default logging level
# Valid values are "trace", "debug", "info", "warn", "error"
level = "debug"

# the destinations of the logging output.
# "-" is for stderr, multiple destinations are possible
default-destinations = [ "-", "/tmp/imag.log" ]

# The format for the logging output
#
# The format supports variables which are inserted for each logging call:
#
#  "%no%"       - The of the logging call
#  "%thread%"   - The thread id from the thread calling the log
#  "%level%"    - The logging level
#  "%module%"   - The module name
#  "%file%"     - The file path where the logging call appeared
#  "%line%"     - The line No of the logging call
#  "%message%"" - The logging message
#
# Functions can be applied to the variables to change the color of
# the substitutions.
#
# A format _must_ contain "%message%, else imag fails because no logging should
# be forbidden
#
[logging.formats]
trace = "cyan([imag][%no%][%thread%][%level%][%module%][%file%][%line%]): %message%"
debug = "cyan([imag][%no%][%thread%][%level%][%module%][%file%][%line%]): %message%"
info  = "[imag]: %message%"
warn  = "red([imag]:) %message%"
error = "red(blinking([imag][uppercase(%level%)]): %message%)"

# Example entry for one imag module
# If a module is not configured or keys are missing
# the default values from above are applied
[logging.modules.libimagstore]
enabled = true
level = "trace"
destinations = [ "-" ]
# A format is only globally configurable, not per-module

One of the most complex things in here would be the format parsing, as variable expansion and functions to apply are some kind of DSL I have to implement. I hope I can do this – maybe there's even a crate for helping me with this? Maybe the shellexpand library will do?

These things and configuration options give the user great power over the logging.

The approach

Because imag already logs a lot, I think about an approach where one thread is used for the actual logging. Because each logging call involves a lot of complexity, I want to move that to a dedicated thread where other threads speak to the logging thread via a MPSC queue.

Of course, this should be opt-in.

The idea is that the logging starts a thread upon construction (which is really early in the imag process, nearly one of the first operations done). This happens when the Runtime object is build and hence no “client code” has to be changed, all changes remain in libimagrt.

This thread is bound to the Runtime object, logging calls (via the logging backend which is implemented for the log crate) talk to it via a channel. The thread then does the heavy lifting. Of course, configuration can be aggregated on construction of the logging thread.

The logging thread is killed when the Runtime object is dropped (one of the last operations in each imag process). Of course, the queue has to be emptied before the logging is closed.

I am also thinking about converting the code base to use the slog crate, which offers structured logging. But I'm not yet sure whether we will benefit from that, because I don't know whether we would need to pass a state-object around. If that is the case, I cannot do this as this would introduce a lot of complexity which I don't want to have. If no such object needs to be passed around, I still have to evaluate whether the slog crate is a nice idea and of course this would also increase the number of (complex) dependencies by one... and I'm not sure whether the benefits outrule the inconveniences.

tags: #linux #open source #programming #rust #software #tools #imag

The imag-pim.org website just got a new face.

I was really eager to do this becaues the old style was... not that optimal (I'm not a web dev, let alone a web designer).

Because the site is now generated using hugo, I also copied the “What's coming up in imag” blog posts over there (I keeping the old ones in this blog for not breaking any links). New articles will be published on the imag-pim.org website.

This very blog article will be published on both sites, of course.

tags: #linux #open source #rust #software #tools #imag

Right now, github shows you this:

And these things will happen more frequently in future, I assure you!

In the first half of 2017, we already had 3 major service outages, 3 minor service outages and 21 other problems. Yes, indeed, this is very good service quality. It really is. Anyways, it is not optimal. Github advertises itself with 22M developers, 59M repositories and 117k businesses world-wide, which is a freakin' lot.

I really like github, it is a great platform for the open-source community and individual projects and developers.

But. There's a but.

Github will not scale endlessly. It will vanish at some point. Maybe not in 2 years, maybe not in 5 or even 10 years. But at some point, a competitor will step up and get bigger and better than github. Or it will be taken down by someone. Who knows.

But when that happens we, as a community, have to be prepared. And that's why we need distributed issue tracking like we implemented with git-dit.

Yes, it is unfortunate that git-dit itself is hosted on github. And we do not have the issues from github in the repository, yet, as there is no mapper available. But we will get there.

I won't go into detail how git-dit works here, there's a talk from GPN17 on youtube about that (in german), where you can learn about these things.

With git-dit, we won't be tied to github and if github vanishes from today to tomorrow, we would be able to continue development seamlessly because the issues are mirrored into the repository itself. In fact, we won't even need github in the first place, because the repository itself would contain everything which is needed for development.

But we are not there yet.

If you're feeling brave, you're more than welcome to try out git-dit or contribute to the codebase.

tags: #git #github #open-source #software #tools