musicmatzes blog

software

I consider myself a power user. I've been using Linux exclusively on all my machines (except maybe on my Androids, if you do not consider these linuxes) for over ten years. Most of the time, I used i3wm (and some time sway), but I got tired of it about two years ago. I switched to XFCE. I told myself, I'm living inside a terminal anyways, I use mutt for mails, (neo)vim for editing stuff and do not use that many GUI apps anyways. The browser, of course. And telegram-desktop. Recently (about a year ago maybe?), riot-desktop joined the two.

I told myself that XFCE is eye-candy enough for me and performs really nice without getting in my way while working in terminals.

Today, I switched my two main devices to KDE Plasma 5. Well, my notebook was actually switched yesterday. And because it was such a huge leap in “niceness”, I switched my workstation today.

I also switched my email setup to Korganizer with Kmail for mails. I am looking forward to see how that performs for me.

I always considered KDE the desktop for Linux. The one, normal users should use to get a first impression of Linux. The one, that will be there for “Linux on the Desktop” when it gets real. You know what I mean?

I never considered for a power-user setup. Power users use i3wm, awesomeWM, sway, ... but not full-blown desktop environments, do they? Well, on several occasions I met Kai Uwe, a KDE dev from a town not far from the town I grew up in. We've met on Linux-Days and other Community events. I always was amazed how fast KDE was on his machines. I was blown away by the usability. But I never really considered switching.

Until today.

Memory usage is where it should be: My desktop uses 3.6GB right now, with Cantata (MPD client), Riot-Desktop (Electron!), Telegram, Firefox and Kontact open (plus some more terminals and some services running in the background as well).

My Notebook was at about 5GB RAM usage with the same apps open and even some more stuff running in the background.

After that experience and also that performance on my rather old desktop machine, I'm blown away and am willing to invest time into getting to know how to be even faster and more productive than I am currently feeling with KDE.

tags: #kde #linux #desktop #software

This post was written during my trip through Iceland and published much later than it was written.

This is a really important topic in programming and I really hope to get this article right. Not only for technical correctness, but also for ease to understand, as explaining types is not that simple if one has never heard of them.

Let's give it a try...

What are types

Well, that's a question which is, in my opinion, not easy to answer. In fact, I thought several days about this question before writing this down, in hope it will become a sufficient answer. Hence, you might find other answers which are easier to understand and maybe more correct as mine, but I'll give it a try nonetheless.

From what I think

Types are a combination of abilities and properties that are combined to express and limit a certain scope of a thing.

For example, A type Car may have four wheels, two doors and a horn (its properties) and can drive slow, drive fast and park (its abilities). That is certainly not a real representation of a car (also because only a car is a real representation of a car) but because of the domain this is used in, it is sufficient in the scenario at hand. The type Car cannot be multiplied, but another type Number may have this ability. Thus, the scope and abilities are also limited in a certain way.

I hope this description is a good one for you to understand.

Now that we know what types are, we should also learn some other terms around the subject of types. The first thing I want to talk about here is “strong typing” and “weak typing”. The reason for this is: These things do not exist. Yes, you've read this correctly: There is no such thing as “strong typing”. There is only stronger and weaker typing. The Java programming language is not strongly typed, neither is it weakly typed (but it is, of course badly typed... forgive me that joke, pity java programmer).

But what is a stronger typing? That is rather simple to explain, actually. We discussed that types are limitations of things to be able to only do some specific operations and such. These things are enforced by the compiler or interpreter of the programming language, of course. And stronger typing only says that the compiler has more information (implicitly via the definition of the programming language) to enforce these rules, the rules of “the function A is defined for type T, so you cannot call it on U”. Of course there is more to that because of generic typing and so on, but that's basically it.

The next term is “type inference”. Type inference is nothing a programmer experiences explicitely, because it happens implicitly. Type inference is a feature of the compiler and interpreter of the language to guess the type of a variable without the programmer stating the actual type. It's nothing more to that actually.

I mentioned the term “generic types” in one of the former paragraphs already, so we should have a look there, too. Generic types, or shorter Generics, are types which are partial, in a way. So for example, one can define a Bag of things, whatever things is. This is often (at least in typed languages – languages where types actually matter for the compiler or interpreter) specified in the code via “type parameters” (though this term differs from language to language).

Why more types are better then few

The more types you introduce in your programs (internally or even for the public API), the more safety you get (speaking in the context of a stronger typed programming language, but also if you do a lot of runtime-type-checking in a weaker typed language). That does not mean that you should introduce a BlueCar, a BlackCar and a GreenCar as types in your program, but rather a type Color and a type Car whereas each Car has a Color – even if your domain is cars and not colors.

Maybe that example lacks a certain expressiveness, so consider this: Your Car has wheels. You can set the number of wheels when constructing the Car object. But instead of passing an integer here, which would yield an API where one can pass 17 as valid number for the number of wheels – or 1337 or possibly even -1. But if you introduce a type which represents the number of wheels, you get some safety into the construction of the Car object – safety checks in your code are not necessary anymore and thus your code will be shorter, better focused on what the actual problem is instead of fighting for valid values and of course, the compiler or interpreter can do the work for you.

Sounds nice, doesn't it? You can get this all with (almost) no cost attached, you just have to write down some more types. If your programming language contains feature like enumerations, you do not even have to make validity checks anymore, as the compiler can execute them.

Next

In the next post we will focus on the coding environment.

tags: #open-source #programming #software #tools #rust

On April 4th, NixOS 18.03 was released.

It is by far the best NixOS release so far, featuring Nix 2.0, major updates for the KDE desktop environment as well as the Gnome Desktop Environment, Linux kernel updated from 4.9 to 4.14 and glibc, gcc and system updated.

With this release, I switched from the unstable channel, which is basically the “rolling release” part of NixOS, to stable (18.03). I did that because I wanted to make sure I get even better stability. Don't get me wrong, even with the unstable channel, I had maybe two or three times in the last year where updating was not possible because one package did not build. But I want to be able to update always, and with 18.03 I get that (at least I hope so).

Also, because as soon as I'm on vacation and possibly without the ability to connect to the internet (or fast internet), I need a certain level of stability. And with the stable channel, the download size for updates shrinks, I guess, because the stable channel only gets small updates and security fixes.

I hope I will be able to switch from 18.03 to 18.09 in October without having too much trouble and downloading the world.

The update/upgrade process was surprisingly simple. The Manual explains what to do and it worked like every other unstable update: Executing some bash commands and then relying on the guarantees NixOS gives me.

Now I'm running stable.

tags: #software #nixos

This post was written during my trip through Iceland and published much latern than it was written.

While we heavily focused on the code-surrounding things in the last parts, we will return to focus on code-related things from here on.

This article discusses code verbosity and how it can improve your open source code and also your contributors experience a lot.

What is code verbosity

Code verbosity is mainly explicitness of code. For example, in Java you have to be more explicit when declaring a variable than in (recent) C++ or even Ruby:

String s = someFunctionCall(param); // Java

auto s = someFunctionCall(param); // C++

s = someFunctionCall param # Ruby

So code verbosity is how explicit you have to state certain things so that the compiler or interpreter understands your intention.

Because we do not always tell the compiler or interpreter what we want to do exactly and because we want to re-use functionality, we introduced abstractions. So abstractions are a way to make code less verbose, in some ways.

How to make code less verbose

Abstraction. It is as simple as this. You introduce abstraction to minimize repetition which leads to less verbose code. Of course, you cannot always make the code less verbose if the language does not allow it: in the above example we used the auto keyword for specifying the type in C++, which is nice, but not possible in Java. So in the borders of your languages abilities, you can make code less verbose.

If you do that right and the abstractions results in nice code, you know that you've done fine.

How much is too much

But there can also be too much abstraction which then yields unreadable code. Not unreadable as in clustered with stuff but just too abstract to grasp at first sight.

Abstraction can get too much. So make sure you introduce sensible abstractions, abstractions that can be combined nicely and of course one can step around the abstractions and use the core functionality, the not-abstracted things beneath.

As a sidenote: sometimes it makes sense to hide certain things completely or even introducing several layers of abstractions.

Next

This was a rather short one, I guess. The next article will be longer I hope, as it will be about typing.

tags: #open-source #programming #software #tools #rust

This post was written during my trip through Iceland and published much latern than it was written.

This is the last post which does not deal with code directly, I promise.

When it comes to open source hobby projects, contributions from others are often happily taken. But making the contribution process smooth for everyone does involve some precautions.

In this article I want to summarize how to make a contribution to a project as smooth as possible for all persons involved.

Public code and contributions

I wrote about this before and I want to shortly reiterate on it. “Open” as in open source (or even better: open contributions) is not black-white at all but there are several levels of grey in between, in my opinion.

The more open your code is, the better a contributor is able to contribute. Whether it be discussion, requests, bug reports, bug fixes or even feature implementation or general enhancement. On the other side, though, the more open your code is the less your contributors are “bound” (in a mentally way) to your project. It can happen (and it happened to me) that a contributor stops by for one pull request or issue and then you'll never hear of them. The better the contribution process is for them, the more likely they come back – and how relevant the project is to them, of course.

The contribution process in the rust community (for the Rust compiler itself) is awesome, from what I've heard. This, of course, enhances the “I will come back and give another issue a try” a lot. The contribution process of the nixpkgs project is slightly worse (but still rather good). Sometimes, nobody answers questions you might have for your pull request for several days or even weeks. This does not really make one eager to file another request.

Platforms

From what I think, github is the “most open” in the sense of “open contributions”. That's not only because of how github works, as other platforms work equally well (gitlab, gitea, gogs, bitbucket) but also because everyone is on github.

If you want to close down contributions a bit, you could host your own instances of gitea or gitlab – contributors can easily open an account for their contribution, though that slight hurdle will make the “casual code dumper” likely go away.

Even “more closed” would be a email-patch-workflow, git supports (and the kernel community uses successfully for years now). In this case, the code is often made available via a web interface like cgit or klaus.

Readme

A project should always contain a readme file in its root folder. The readme file is often the first thing a contributor will look at, not only but also because github renders and displays them.

Therefore, keeping your readme file up to date and filled with current information can be a good way to show your contributors (and users) what is going on in the projects code base. It should contain a short description what the project/code does and how it works (only from a users view – implementation details or why you implemented this in Haskell instead of JavaScript do not necessarily belong here). It should contain a few examples how to use the code or, if it is a library, how to call it. It also should contain build instructions (if necessary) or a pointer to a “BUILDING” file if the build process is long or complicated. At the end of the Readme file, a license statement (how the project is licensed) should be pasted. Not the entire license, but only a short note and a copyright note as well. It happens to be kind to do so.

Contributing File

Often, projects contain a Contributing file where guidelines (or even rules) are written down on how to contribute. It does not only contain statements on how code is submitted but also how issues are filed or requests should be made.

I think it is extremely important to have such an file available, especially if not hosting the code on a site like github, where it is obvious that code is submitted through pull requests and issues are submitted via the issue tracker.

The length of such an file should respect the size of the project itself. If the project contains 10KLOC, one should be able to read the contribution file in less than two minutes, preferably in less than one minute. It should state not only how code should be submitted, but also whether it should conform to some style guide (which itself can be outsourced to yet another file), how to behave in the community, how to write bug reports and also how to file issues (what information must be included).

Issue handling

Handling issues is clearly a way to improve the contributors experience. As soon as a contributor files an issue, she or he should be greeted and thanked for the issue. Take it this way: Someone just invested time to look at your project and cared enough to have a question, try it out or even found a bug. This is truly a cool thing and therefore they should be thanked for this, as soon as you have the time to do so. The Rust community even automated this, but I don't think this would be necessary for a small or medium sized project/community.

So be nice to every one. Nothing is worse than a maintainer that babbles about bad things or insults the contributor because of his or her ideas or ways an issue was proposed to be resolved. Don't ever do this. I've seen issues where the maintainer of the project started rambling about how bad things were (not the project itself but rather its dependencies or even things that had nothing to do with the project itself). I cannot believe that such projects will last long, let alone survive at all. These projects will die.

Also, your ramblings have nothing to do with the issue at hand and even if they do: be kind and humble will most likely be better in every way, right?

Next

In the next part we will finally go back and actually talk code.

What I want to discuss in the next article of this series is code verbosity. I want to make sure how DRY a code actually needs to be and how much abstraction is enough for the sake of understandability and cleanness of code.

tags: #open-source #programming #software #tools #rust

I love the Rust language. And I love the library ecosystem the Rust community provides.

But let me start at the beginning.

libgitdit

In 2016, I and a friend of mine developed a library for distributed issue tracking with git and a commandline frontend for that library. The project, git-dit got a rather good portion of attention on hackernews, reddit and of course on github. Ranking at about 350 stars on github, this is the most successfull piece of software I (co-)authored so far.

Development sleeps right now, which is really unfortunate. There is a number of unresolved issues, despite the software is usable today.

A failed thesis

My friend and I proposed a bachelors thesis at our university for writing a web-viewer for git-dit. Because we're both master students, we also offered to supervise this thesis.

In the last semester, the thesis was assigned and I was happy when it started. Today (or rather a few days ago) I was not that happy anymore, because I got the end-result. I was able to compile it (yay), but after starting it, I was not even able to open the web page because I did not know which port to use.

After looking at the source, I was able to figure that part out. Unfortunately, everything I tried to do via the web frontend failed (as in nothing happened). I was not able to see any issues or anything else. Only viewing the git configuration was possible – but that's the least thing I cared about.

So I figured: How hard can it be? If a bachelor student has half a year time,... it must be hard? No, I guess not.

Lets do that!

So I started my own git-dit-web viewer. And I tracked the time it took to implement it.

I mean, how hard can it be? I am not a web-dev at all, I have zero experience with Rust web frameworks, I never touched one. I have no experience with CSS (only that view bits I used for this blog) and of course I also have no experience with JS.

But how hard can it be?

Turns out: It is not hard at all. I'm proud to present the first prototype, after 11 hours of implementation time:

$ time sum :week git-dit-wui

Wk  Date       Day Tags           Start      End    Time    Total
--- ---------- --- ----------- -------- -------- ------- --------
W11 2018-03-15 Thu git-dit-wui 14:00:00 16:43:35 2:43:35
                   git-dit-wui 20:31:00 23:52:04 3:21:04  6:04:39
W11 2018-03-16 Fri git-dit-wui 11:23:39 14:12:56 2:49:17
                   git-dit-wui 15:45:16 17:58:47 2:13:31  5:02:48

                                                         11:07:27

What does not work yet

Of course, this is only the first prototype. The following things do not work yet:

  • Filtering issues for open/closed or other metadata
  • Showing issues which were opened by one specific author
  • Show messages as tree (currently linear by timestamp)
  • Graph for issues-per-date (nice-to-have)
  • Showing commits
  • Automatically detecting git hashes in messages and linking to the appropriate issue/commit
  • Abbreviating git hashes in messages and everywhere else
  • Configuration
    • Port
    • Repository path
    • Readonly/RW
  • Error handling: if things go wrong, we should show an error page rather than nothing

What does work

But some things also work, of course:

  • Messages are rendered as markdown
  • Listing all issues (with some metadata)
  • Showing an issue (including replies)
  • Showing single messages
  • Landing page with statistics about issues

And it looks rather good (thanks to the bulma CSS framework) despite me beeing a CLI-only guy without web-dev experience.

Screenshots

Some screenshots showing the issues in the git-dit repository.

The landing page The issue listing page Showing a single issue

Conclusion

Of course I open-sourced the code on github and licensed it as AGPL-3.0.

So it can be done. I'm not quite sure what the student did in 6 months time he had for implementing this.

tags: #network #open-source #software #git

After thinking a while about the points I layed out in my previous post I'd like to update my ideas here.

It is not necessary to read the first post to understand what I am talking about in this second one, but it also does not do any harm.

Matrix and Mastodon are nice – but federation is only the first step – we have to go towards fully distributed applications!

(me, at the 34. Chaos Communication Congress 2017)

The idea

With the rise of protocols like the matrix protocol, activitypub and others, decentralized social community platforms like matrix, mastodon and others gained power and were made real. I consider these platforms, especially mastodon and matrix, to be great steps into the future and am using both enthusiastically. But I think we can do better. Federation is the first step out of centralization and definitively a good one. But we have to push further - towards full distributed environments!

(For a “Why?” have a look at the end of the article!)

How would it work?

The foundations how a social network on IPFS would work are rather simple. I am very tempted to use the un-word “blockchain” in this article, but because of the hype around that word and because nobody really understands what a “blockchain” actually is, I refrain from using it.

I use a better one instead: “DAG” – “Directed Acyclic Graph”. Also “Merkle-Tree” is a term which could be used, but when using this term, a notion of implementation-details comes to mind and I want to avoid that. One instantly thinks of crypto, hash values and blobs when talking about hash trees or merkle trees. A DAG though is a bit more abstract concept which fits my ideas better.

What we would need to develop a social network (its core functionality) on IPFS is a DAG and some standard data formats we agree upon. We also need a private-public-key infrastructure, which IPFS already has.

There are two “kinds” of data which must be considered: meta-data (which should be replicated by as many nodes as possible) and actual user data (posts, messages, images, videos, files). I'm not talking about the second one here very much, because the meta-data is where the problems are.

Consider the following metadata blob:

{
  "version": 1,
  "previous": [ "Qm...1234567890" ],

  "profile": [ "Qm...098765", "Qm...54312" ],

  "post": {
    "mimetype": "text/plain",
    "nodes": ["Qm...abc"],
    "date": "2018-01-02T03:04:05+0200",
    "reply": [],
  },

  "publicfollow": [ "Qm...efg", "Qm...hij" ]
}
  • The version key describes the version of the protocol, of course.

  • Here, the previous array points to the previous metadata blob(s). We need multiple entries here (an array) because we want to create a DAG.

  • The profile key holds a list of IPNS names which are associated with the profile.

The version, previous and profile keys are the only ones required in such a metadata blob. All other keys shown above are optional, though one metadata-blob should only contain one at a time (or none).

  • The post table describes the actual userdata. Some meta-information is added, for example the mimetype ("text/plain" in this case) and the date it was created. More can be thought of. The nodes key points to a list of actual content (again via IPFS hashes). I'm not yet convinced whether this shall be a list or a single value. Details! I'd say that these three keys are required in a post table. The reply key notes that this post is a reply to another post. This is, of course, optional.

  • The publicfollow is a list of IPNS hashes to other profiles which the user follows publicly. Whether such a thing is desireable is to be discussed. I show it here to give a hint on the possibilities.

  • More such data could be considered, though the meta-data blobs should be held small: If one thinks of 4kb per meta-data blob (which is a lot) and 10 million blobs (which I do not consider that much, because every interaction which is a input into the network in one form or another results in a new meta-data blob), we have roughly 38 GB of meta-data content, which is really too much. If we have 250 bytes per metadata-blob (which sounds like a reasonable size) we get 2.3 GB of meta-data for 10 million blobs. That sounds much better.

The profile DAG

The idea of linking the previous version of a profile from each new version of the profile is of course one of the key points. With this approach, nobody has to fetch the whole list of profile versions. Traversing the whole chain backwards is only required if a user wants to see old content from the profile she's browsing.

Because of IPFS and its caching, content automatically gets replicated over nodes as users browse profiles. Nodes can cache either only meta-data blobs (not so much data) or user content as well (more data). This can happen automatically or user-driven – several possibilities here! It is even possible that users “pin” content if they think its important to keep it.

Profile updates can even be “announced” using PubSub so other nodes can then fetch the new profile versions and cache them. The latest profile metadata-blob (or “version”) can be published via a IPNS name. The IPNS name should be published per-device and not per-account. (This is also why there is a devices array in the metadata JSON blob!)

Why should we publish IPNS names per-device and why do we actually need a DAG here? That's actually because of we want multi-device support!

Multi-device support

I already mentioned that the profile-chain would be a DAG. I also mentioned that there would be a profile key in the meta-data blob.

This is because of the multi-device support. If two, three or even more devices need to post to one account, we need to be able to merge different versions of an account: Consider Alice and Bob sharing one account (which would be possible!). Now, Bob loses connection to the internet. But because we are on IPFS and work offline, this is not a problem. Alice and Bob could continue creating content and thus new profile versions:

A <--- B <--- C <--- D <--- E
        \
         C' <--- D' <--- E'

In the shown DAG, Alice posts C, D and E, each referring to the former. Bob creates C', D' and E' – each refering to the former. Of course both C and C' would refer to B.

As soon as Bob comes back online, Alice notices that there is another chain of posts to the profile and can now merge the chains be publishing a new version F which points to both E and E':

A <--- B <--- C <--- D <--- E <--- F
        \                         /
         C' <--- D' <--- E' <-----

Because Bob would also see another chain, his client would also provide a new version of the profile (F') where E and E' are merged – one of the problem which must be sorted out. But a rather trivial one in my opinion, as the clients need only to do some sort of leader-election. And this election is temporary until a new node is published – so not really a complicated form of concensus-finding!

What has to be sorted out, though, is that the devices/nodes which share an account and now need to agree upon which one merges the chains need some form of communication between them. I have not yet thought about how this should be done. Maybe IPFS PubSub is a viable option for this. Cryptographic signatures play a important role here.

This gets a bit more complicated if there are more than two devices posting to one account and also if some of them are not available yet – though it is still in a problem space near “we have to think hard about this” ... and nowhere in the space of “seems impossible”!

The profile key is provided in the account data so the client knows which other chains should be checked and merged. Thus, only nodes which are already allowed to publish new profile versions are actually allowed to add new nodes to that list.

Deleting content in the DAG

Deleting old versions of the profile – or old content – is possible, too. Because the previous key is an array, we can refer to multiple old versions of a profile.

Consider the following chain of profile versions:

A<---B<---C<---D<---E

Now, the user wants to drop profile version C. This is possible by creating a new profile version which refers to E and B in the previous field and then dropping C. The following chain (DAG) is the result:

A<---B     <---D<---E<---F
      \                 /
       -----------------

Of course, D would now point to a node which does not exist. But that is not a problem. Indeed, its a fundamental key point of the idea – that content may be unavailable.

F should not contain new content. If F would contain new content, dropping this content would become harder as the previous key would be copied over, creating even more links to previous versions in the new profile version.

“Forgetting” content

Because clients won't traverse the whole chain of a profile, but only the newest 10, 100 or 1,000 entries, older content gets “forgotten” slowly. Of course it is still there and the device hosting it still has it (and other devices which post to the same account, eventually also caching servers). Either way, content gets forgotten slowly. If the user who published the content deletes it, the network may be unable to fetch it at some point.

Is that bad? I don't think so! Important content gets replicated by others, so if I post a comment on an article, I could (automatically or manually) pin the article itself in my IPFS instance to preserve it. If I do not and the author of the article thinks that it might not be that interesting, the article may be deleted and gets unavailable to the network.

And I think that is fine. Replicate important content, delete unimportant content. The user has the power to decide here!

Comments on posts (and comments)

Consider you want to comment on a post. Of course you create new content, which links to the post you just commented. But the person who wrote the original post does not automatically link to your comment, so nobody is able to find your comment.

The approach for solving this is to provide updates to content. An update is simply a new meta-data blob in the profile. The blob would contain a link to the original post and the comment on it:

{
  "version:" 1,
  "previous": [ "Qm...1234567890" ],

  "profile": [ "Qm...098765", "Qm...54312" ],

  "post": {
    "update": "Qm...abc",
    "new-reply": "Qm...ghjjk",
  },
}

The post.update and post.new-reply would link to meta-data blobs: The update one to the original post or the latest update on the post – the new-reply one on the post from the other user which provides a comment on the post. Maybe it would also be an option to list all direct replies to the post here. Details!

Because this works on both “posts” and “reply” kind of data, comments on comments are possible.

Comments deep down the chain of comments would have to slowly propagate to the top – to the actual post.

Here, several configurations are possible:

  • Automatically include comments and publish new profile versions for them
  • Publishing/propagating comments until some mark is hit (original post is more than 1 month old, more than 100 comments are propagated)
  • User can select other users where comments are automatically propagated and others have to be moderated
  • User has to confirm propagation (moderated comments).

The key difference to known approaches here is that not the author of the original post permits comments, but always the author of the post or comment the reply was filed for. I don't know whether this is a nice thing or a problem.

Unavailable content

The implementation of the social network has to be error-resistant, of course. IPFS hashes might not be there, fetching content might not be possible (temporarily or at all). But that's an implementation-detail to me and I will not lose any more words about it.

Federated component

One might think “If I go offline with my node, my posts are not accessible if nobody else is online having them”. And that's true.

That's why I would introduce a federated component, which would run a stripped-down version of the application.

As soon as another instance connects and a new post is announced, the instance automatically pins or caches it. Of course, this would mean that all of these federated instances would pin all content, which is surely not nice. Posts which are pinned for a certain amount of time are most likely distributed well enough so the federated component nodes can drop them... maybe after 90 days, maybe after 10... Details!

Subscribing (privately) and other private information

Another issue with multi-device support would be subscribing privately to another account. For example, if a user (lets call her Amy) subscribes to another user (lets call him Sheldon) on her Notebook, this information needs to be stored somehow. And because Amys machines do not necessarily sync with each other, her mobile phone may never know that following Sheldon is a thing now!

This problem could by solved by storing the “follow”-information in her public profile. Although, some users might not like everyone to know who to follow. Cryptographic things could be considered to fix visibility.

But then, users may want to “categorize” their friends, store them in groups or whatever. This information would be stored in the public profile as well, which would create even more noise on the network. Also, because cryptography is hard and information would be stored forever, this might not be an option as some day, the crypto might be broken and reveal all the things that were stored privately before.

Another solution for this would be that Amys devices would have to somehow sync directly, without others beeing able to read any of that data. Something like a CRDT which holds a configuration file which is then shared between the devices directly (think of a git-repository which is pushed between the devices directly without accessing a server on the internet). This would, of course, only work if the devices are on the same network.

As you see, I have not thought about this particular problem very much yet.

Discovering content

What I did not spend much time thinking about as well was how clients discover new content. When a user installs a client, this client does not know any IPFS peers – or rather any “social network nodes” where it can fetch user profiles/data from - yet. Even if it knows some bootstrap nodes to connect to, it might not get content from them if they do not serve any social network data and if the user does not know any hashes of user profiles. To be able to find new social network IPFS nodes, a client has to know their IPNS hashes – But how to discover them?

This is a hard problem. My first idea would be a PubSub channel where each client periodically announces their IPNS hashes. I'm not sure whether PubSub nodes must be connected directly. If this is the case, the problem just got harder. There's also the problem that this channel would be rather high-volume as soon as the network grows. If each client announces their IPNS hash every 15 minutes, for example, we get 4 messages per client each hour. That's already a lot of bandwidth if we speak about 1,000, 10,000 or even 100,000 clients! It is also an attack-vector how the system can be flooded. Not nice!

One way to think about this is that if only nodes which are subscribed to a topic do also forward the topics messages (like this comment suggests), we could reduce the time between “publishing” messages in the topic. Such a message would contain all IPNS hashes a node knows about, thus the amount of data would be rather much. As soon as the network grows, a node would need to send this message less and less often, to reduce the number of messages and bytes send. Still, if each node knows 10,000 nodes and sends this list once an hour, we get

bytes_per_hash = 46
number_of_nodes = 10_000
message_size = bytes_per_hash * number_of_nodes
bytes_per_hour = number_of_nodes * message_size

4,28 GiB of “I know these nodes” messages per hour. That does obviousely not scale!

Maybe each client should offer an API where other clients can ask them about which IPNS hashes they know. That would be a “pull” approach rather than a “push” approach then, which would limit bandwidth a bit. This could even be done via PubSub as well, where the channel name is generared from the IPFS instance hash, for example. I don't know whether this would be a nice idea. Still, this would need some “internet-facing” software where clients need to be able to talk directly to eachother. I don't know whether IPFS offers functionality to do this in a simple way.

Either way, I have no solution for this problem yet.

Why IPFS?

Platforms like scuttlebutt or movim also implement distributed social networks, why not use those? Also, why IPFS and not the dat protocol or something else?

That question is rather simple to answer: IPFS provides functionality and semantics other tools/frameworks do not provide. Most importantly the notion that content is immutable, but also full decentralization (not federation like with services like movim or mastodon, for example).

Having immutable content is a key point. The dat protocol, for example, features mutable content as it is roughly based on bittorrent (if I understood everything correctly, feel free to point out mistakes). That might be nice in some cases, though I think immutability is the way to go. Distributed applications or frameworks for distributed content with immutability as core concept are better suited for netsplit, slow connections and peer-to-peer applications. From what I saw from the last weeks and months of looking at frameworks for distributed content storage is that IPFS is way more mature than these other frameworks. IPFS is build to replace existing contents and to stay, and that's a nice thing to build applications on.

Remaining Questions

Some questions are remaining:

  • Is it possible to talk to a specific node directly in IPFS? This would be helpful for discovering content by asking nodes what profiles they know. It would also be a nice way for finding consensus when multiple devices have to agree on which node publishes a merge.
  • How fast is IPFS with small files? If I need to traverse a long chain of profile updates, I constantly request a small file, parse it and continue requesting the previous node in the chain. That should be fast. If it is not, we might need to introduce some “pack files” where a list of metadata-nodes is provided and traversing becomes unnecessary with. But that makes deleting content rather complicated, TBH.

That's all I can think of right now, but there might be more questions which are not yet answered.

Problems are hard in distributed environments

Distributed systems involve a lot of new complexity where we have to carefully think about details and how to design our system. New ways to design systems can be discovered by the “distributed approach” and new paradigms emerge.

Moving away from a central authority which holds the truth, the global state and also the data results in a paradigm shift we really have to be careful about.

I think we can do it and design new, powerful and fully distributed systems with user freedom, usability, user-convenience and state-of-the-art in mind. Users want to have a system which is reliable, failure proof, convenient and easy to use. They want to “always be connected”. I think we can provide such software. Developers want nice abstractions to build upon, data integrity, failure-proof software with simplicity designed into the system and reusable data structures and be able to scale. I think IPFS is the way to for this. In addition, I think we can provide free software with free data.

I do not claim to know the final solution to any of the problems layed out in this article. Its just that I think of them and would love to get an open conversation started on the whole subject of distributed social networks and problems that come with them.

And maybe we can come up with a prototype for this?

tags: #distributed #network #open-source #social #software

Why can't we do

fn foo(i: i32, j: i32) -> i32 {
    i + j
}

fn main() {
    println!("{}", foo((1, 2)));
}

in Rust?

The idea

First of all: I don't know much about what is going on in rust-compiler-land, hence I don't know whether this is technically possible or even in progress already. I'm simply writing down some thoughts here.

The idea, or rather the question, is: Why can't we map tuples on arguments in Rust? The code from above shows exactly what I mean by that: Why is the compiler not able to unpack the tuple and pass it as list of arguments to the function?

It would be really cool if it could do that. The usecase is something like this:

some_possibly_failing_fn() // -> Result<_, _>
  .and_then(|res| (res, some_other_failing_fn()?)) // Result<(_, _), _>
  .map(PartialEq::Eq) // Result<bool, _>

for example. If mapping of tuples on arguments would be possible, one could write code like this, where failing functions are chained and operators and functions can then be called on the results of the failing functions.

Currently, one has to write it like this:

some_possibly_failing_fn() // -> Result<_, _>
  .and_then(|res| (res, some_other_failing_fn()?)) // Result<(_, _), _>
  .map(|(a, b)| a == b) // Result<bool, _>

or, because it is solvable with one line less:

some_possibly_failing_fn() // -> Result<_, _>
  .and_then(|res| Ok(res == some_other_failing_fn()?)) // Result<bool, _>

You might think that the overhead for that convenience is rather low and thus it is not really beneficial to have such a functionality in the rust compiler.

That is true. But there are great possibilities of API design if this is mapping of tuples on arguments are possible. For example, one could write an extension to the Result type which extends tuples. Think of something like that:

// assuming:
//
// some_possibly_failing_fn() -> Result<T, _>;
// some_other_failing_fn()) -> Result<U, _>;
// even_other_failing_fn()) -> Result<W, _>;
// yet_another_failing_f()) -> Result<V, _>;
// check_results(T, U, W, V) -> Result<bool, Error>;
//

some_possibly_failing_fn() // -> Result<T, _>
  .and_then_chain(|| some_other_failing_fn()) // Result<(T, U), _>
  .and_then_chain(|| even_other_failing_fn()) // Result<(T, U, W), _>
  .and_then_chain(|| yet_another_failing_f()) // Result<(T, U, W, V), _>
  .and_then(check_results)

From an API-design standpoint, this is plain awesome.

Lets compare that to what we have to write today to do this:

some_possibly_failing_fn() // -> Result<T, _>
  .and_then(|a| Ok((a, some_other_failing_fn()?)))
  .and_then_chain(|(a, b)| Ok((a, b, even_other_failing_fn()?)))
  .and_then_chain(|(a, b, c)| Ok((a, b, c, yet_another_failing_f()?)))
  .and_then(|(a, b, c, d)| check_results(a, b, c, d))

and that's really not that readable.

From a technical point of view, the compiler is able to verify the length of the tuples and thus is able to check whether the tuple actually matches the number of parameters of a function.

As said, I'm not that much into the development of the language, thus I don't know whether there are other things that prevent this, though.

Update

After posting this as a question on users.rust-lang.org and getting some nice replies with explanations why this cannot be, I conclude:

It is not possible to do it in a way that does not yield weird edge cases, ugly implicit conversions and other weirdness we really do not want in the language.

But I learned something, and that was worth it!

tags: #tools #software #c++

Today I learned that the new C++ standard (C++17) has std::variant – but no pattern matching.

Variants and how they work

Variants or “tagged unions” are a special form of data type. They describe an abstract thing which could be one concrete thing or another concrete thing. Variants are available in a lot of languages, including and not limited to ML, Haskell, Algol 68, Pascal, Ada, Rust and even C.

With C++17, the C++ standard was extended to have the std::variant generic type, which is basically a “C union on steroids”.

(To the C++ greybeard programmers reading this: I know this is technically not right, how it is implemented and how this works is very different from an actual C union, but what it is used for is basically the same, so please don't yell at me for this statement.)

A std::variant can be used to express, for example, a return value of a function which might have failed. Maybe you think “Hey, that's what exceptions are for” – no, that's not what exceptions are for, though a lot of programmer use them like this. Exceptions are for unexpected behaviour of your program. If you're writing a terminal calculator, an invalid input is an expected failure. An unexpected failure would be that you cannot read from stdin!

In cases where expected failures happen, a std::variant can be used to return a “I failed doing my job”-value.

Why you need pattern matching

Pattern matching can be used to destructure variants efficiently (as in code, readability, extendability and thus maintainability) to access their inner values.

For example, in Haskell, one can destructure a variant like this:

data Result = Ok | Err

explainresult :: Result -> [Char]
explainresult Ok  = "All right"
explainresult Err = "Ouch"

Here, we define the variant Result and a function explainresult which destructures the variant and returns a string based on the passed variant. Of course, variants can hold data and this data can be accessed then.

As a matter of fact, variants are incredible useful in everyday programming and so is pattern matching for destructuring them.

What C++17 offers instead of pattern matching

C++17 does not offer pattern matching. Why? I don't know. But it really should have, because the way this issue is solved in C++17 is painful: std::visit is used for imitate pattern matching.

On the first glance, the following code looks rather okayish (well, it is still C++, right?):

std::vector<std::variant<char, long, float, int, double, long long>>
           vec = {5, '2', 5.4, 100ll, 2011l, 3.5f, 2017};

// display each value
for (auto& v: vec){
  std::visit([](auto&& e){std::cout << e << " ";}, v);
}

But this is simple. What if we want to actually do something with the element based on which type it has?

Well, not so nice: We have to design our own visitor and equip it with the functionality:

struct Visitor {
    void operator()(const string& s) const {
      // ...
    }

    void operator()(const int n) const {
      // ...
    }
};

While having a dedicated visitor which can be passed around is rather nice, having it as the only option for pattern matching is hell.

Also, the standard library does not provide a helper for building a visitor object from a list of lambdas!

All in all: Is the whole thing a step into the right direction? No. It is rather a stumble into the rightish direction. Nothing more.

A friendly advice from a non-Cpp programmer

Take a grain of salt. I'm not a (C++) professional, hence this might not reflect the opinions of the broader community. Also, the following might get some people really angry. But I don't care.

Don't use C++ for new projects!

I know C++ is a heavy target and the standard is backwards compatible and a lot of people count on the standards committee to get this right and yaddayaddayadda. From my point of view, C++ tries to catch up. This is, of course, nice. But at that speed, other languages will innovate much faster and better and C++ will slowly die. That's one reason I would never ever use C++ for a new project.

tags: #tools #software #c++

For one of my other projects (yes, imag), I developed a query library for TOML. I currently am planning a new feature for it: A query API which can be used to execute prepared queries on a given TOML document.

But let me start with the basics!

What is TOML?

“TOML” stands for “Tom's Obvious, Minimal Language” and is somewhat similar to JSON, though is highly readable and easy to understand. It is mainly used for configuration files in the Rust community, I use it as header format for imag store entries and as configuration file format for imag, for example.

A mini TOML file looks like this:

[table]
key = "value"
[table.subtable]
key = ["another value in an array"]

What does 'toml-query' do?

toml-query, the library I developed, is actually an extension. It extends the toml-rs library, which is a serde based library for the TOML data format. Serde is the serialization/deserialization framework in the rust ecosystem. Thus, toml-rs is a frontend to that framework to work with the file format.

Because serde is such an amazing tool, one can write code like this:

[derive(Serialize, Deserialize)]
struct Foo {
  integer: i32,
  string: String
}

to get a struct which can be serialized to and deserialized from TOML with minimal effort in Rust:

extern crate serde;
#[macro_use] extern crate serde_derive;
extern crate toml;

#[derive(Serialize, Deserialize)]
struct Foo {
    integer: i32,
    string: String,
}

fn main() {
    let foo = Foo {
        integer: 15,
        string: String::from("Hello world!"),
    };

    let serialized = toml::to_string(&foo).unwrap(); // here is magic!
    let text = r#"integer = 15
string = "Hello world!"
"#;
    assert_eq!(text, serialized);
}

(this piece of code can be executed with the playground).

The resulting TOML can, of course, be deserialized back to an instance of Foo. That's really neat if you want to read your configuration file, because you simply have to write a struct which describes the variables your configuration file should have and let toml-rs and serde do the magic of failure-free deserialization. If an error happens, for example a key is not there, the deserialization fails and you can forward the error to your user, for example.

But what happens if you have a really complex configuration file? What if you don't know, at build time of your program, what your configuration file looks like? What if you have things that are allowed to go wrong and you have to very precisely catch errors and handle them individually? Then, this awesomeness becomes complicated.

That's why I wrote toml-query. It helps you maintain a real CRUD (Create-Read-Update-Delete) workflow on TOML documents. For example, when reading your toml document into memory and into toml-rs structures, you can then read and write specific values by their path:

extern crate serde;
extern crate toml;
extern crate toml_query;

fn main() {
    let text = r#"integer = 15
string = "Hello world!"
"#;
    let toml : toml::Value = toml::de::from_str(text).unwrap();
    let int = toml.read("integer") {
        Ok(Some(&Value::Integer(i))) => i,
        Ok(Some(_)) => panic!("Type error: Not an integer!"),
        Ok(None)    => panic!("Key 'integer' missing"),
        Err(e)      => panic!("Error reading TOML document: {:?}", e);
    }
}

The upper code example reads the TOML document into a Value (which is a datatype provided by toml-rs) and then read()s the value at "integer". This read operation is done via the “path” to the value, and of course this path is not only a string. Things like "table.subtable.value" are possible. Array indexes are possible. This works with several CRUD operations: Reading values, writing values and creating intermediate “tables” or “arrays” if they are not already created, updating values and of course also deleting values.

Why a Query-API?

The things I explained above are entirely CRUD functionality things. There is no “query” thing here.

The next step I am currently thinking about is an API which can be used to build complex queries, chaining them and (not in the first version of the API, but maybe later), also rolling them back.

The idea would be an API like this:

let query = Read::new("foo")
  .and_then(SetTo::new(|x| x + 1))
  .and_then(DeleteAt::new("bar"));

let query_result = document.execute(query);

Here, we build a query which reads a value at “foo”, then increments that value and after that deletes the value at “bar”. If one of these steps fails, the others are never executed.

The equivalent in CRUD calls would look like this:

let value = document.read("foo").unwrap().unwrap();
let value = value + 1;
document.set("foo", value).unwrap();
document.delete("bar").unwrap();

The calls to unwrap() are here to show where errors can happen. All this would be hidden in the query functionality and the query_result would then hold an error which can be used to tell the user what exactly went wrong and where.

The exact shape and semantics of the API are not as shown in the example above. The example is solely used for visualizing how the API would look like.

How does the Query-API work?

The basic idea here is to encapsulate CRUD calls into objects which then can be chained in some way. That's the whole thing, actually.

The important thing is that the user is able to define own “Query types”: Types which can be put into the chain of queries and which are composed of other query types. This way, a user can basically define structures and procedures to write code like this:

let result = document.execute(TransformConfigFileFormat::new());

and the TransformConfigFileFormat type then transforms an old config file format to a new one (for example).

This requirement makes the whole thing complicated. To list the full requirements:

  • A query may return data which may be used by the next query in the chain
  • A user of the library should be able to compose new query objects from existing ones
  • The CRUD functionalities shall be provided as “query types”
  • The API should be easy to use with both “query types” and closures (think: query = other_query.and_then(|val| val + 1);

Reversible queries / Transactions

A really nice thing would be reversible queries.

In this scenario, one would be able to call a chain of queries and if one of the queries fails, the document is left untouched. This could be done by either copying the whole document before executing the query-chain and replacing the modified version with the unmodified if something failed, or by making the queries actually role-back-able (thus, an insert would reverse to a delete and the other way round, for example).

The first idea is more memory-intensive and the latter more runtime/CPU intensive. Maybe both would be an idea and the user is then able to decide.

Other things

One other thing which would be really great is to generalize the functionality of toml-query over all data-formats serde provides serialization and deserialization functionality for.

This would be the ultimate end-game and I'm sure I'm not able to do this without help (because toml-query is already really complex right now and such a thing would increase complexity even more).

If someone wants to step up and do this, I'd love to help!

tags: #software #rust #open-source