musicmatzes blog

Today, I wrote a mastodon bot.

Shut up and show me the code!

Here you go.

The idea

My idea was, to write a bot that fetches the lastest master from a git repository and counts some commits and then posts a message to mastodon about what it counted.

Because I always complain about people pushing to the master branch of a big community git repository directly, I decided that this would be a perfect fit.

(Whether pushing to master directly is okay and when it is not okay to do this is another topic and I won't discuss this here)

The dependencies

Well, because I didn't want to implement everything myself, I started pulling in some dependencies:

  • log and env_logger for logging
  • structopt, toml, serde and config for argument parsing and config reading
  • anyhow because I don't care too much about error handling. It just has to work
  • getset for a bit cleaner code (not strictly necessary, tbh)
  • handlebars for templating the status message that will be posted
  • elefren as mastodon API crate
  • git2 for working with the git repository which the bot posts about

The Plan

How the bot should work was rather clear from the outset. First of all, it shouldn't be a always-running-process. I wanted it to be as simple as possible, thus, triggering it via a systemd-timer should suffice. Next, it should only fetch the latest commits, so it should be able to work on a working clone of a repository. This way, we don't need another clone of a potentially huge repository on our disk. The path of the repository should of course not be hardcoded, as shouldn't the “upstream” remote name or the “master” branch name (because you might want to track a “release-xyz” branch or because “master” was renamed to something else).

Also, it should be configurable how many hours of commits should be checked. Maybe the user wants to run this bot once a day, maybe once a week. Both is possible, of course. But if the user runs it once a day, they want to check only the commits of the last 24 hours. If they run it once a week, the last 168 hours would be more appropriate.

The message that gets posted should also not be hardcoded, but a template where the variables the bot counted are available.

All the above goes into the configuration file the bot ready (and which can be set via the --config option on the bots CLI).

The configuration struct for the setup described above is rather trivial, as is the CLI setup.

The setup

The first things the bot has to do is reading the commandline and the configuration after initializing the logger, which is a no-brainer, too:

fn main() -> Result<()> {
    env_logger::init();
    log::debug!("Logger initialized");

    let opts = Opts::from_args_safe()?;
    let config: Conf = {
        let mut config = ::config::Config::default();

        config
            .merge(::config::File::from(opts.config().to_path_buf()))?
            .merge(::config::Environment::with_prefix("COMBOT"))?;
        config.try_into()?
    };
    let mastodon_data: elefren::Data = toml::de::from_str(&std::fs::read_to_string(config.mastodon_data())?)?;

The mastodon data is read from a configuration file that is different from the main configuration file, because it may contain sensitive data and if a user wants to put their configuration of the bot into a (public?) git repository, they might not want to include this data. That's why I opted for another file here, its format is described in the configuration example file (next to the setting where the file actually is).

Next, the mastodon client has to be setup and the repository has to be opened:

    let client = elefren::Mastodon::from(mastodon_data);
    let status_language = elefren::Language::from_639_1(config.status_language())
        .ok_or_else(|| anyhow!("Could not parse status language code: {}", config.status_language()))?;
    log::debug!("config parsed");

    let repo = git2::Repository::open(config.repository_path())?;
    log::debug!("Repo opened successfully");

which is rather trivial, too.

The Calculations

Then, we fetch the appropriate remote branch and count the commits:

    let _ = fetch_main_remote(&repo, &config)?;
    log::debug!("Main branch fetched successfully");

    let (commits, merges, nonmerges) = count_commits_on_main_branch(&repo, &config)?;
    log::debug!("Counted commits successfully");

    log::info!("Commits    = {}", commits);
    log::info!("Merges     = {}", merges);
    log::info!("Non-Merges = {}", nonmerges);

The functions called in this snippet will be described later on. Just consider them working for now, and let's move on to the status posting part of the bot now.

First of all, we use the variables to compute the status message using the template from the configuration file.

    {
        let status_text = {
            let mut hb = handlebars::Handlebars::new();
            hb.register_template_string("status", config.status_template())?;
            let mut data = std::collections::BTreeMap::new();
            data.insert("commits", commits);
            data.insert("merges", merges);
            data.insert("nonmerges", nonmerges);
            hb.render("status", &data)?
        };

Handlebars is a perfect fit for that job, as it is rather trivial to use, albeit a very powerful templating language is used. The user could, for example, even add some conditions to their template, like if there are no commits at all, the status message could just say “I'm a lonely bot, because nobody commits to master these days...” or something like that.

Next, we build the status object we pass to mastodon, and post it.

        let status = elefren::StatusBuilder::new()
            .status(status_text)
            .language(status_language)
            .build()
            .expect("Failed to build status");

        let status = client.new_status(status)
            .expect("Failed to post status");
        if let Some(url) = status.url.as_ref() {
            log::info!("Status posted: {}", url);
        } else {
            log::info!("Status posted, no url");
        }
        log::debug!("New status = {:?}", status);
    }

    Ok(())
} // main()

Some logging is added as well, of course.

And that's the whole main function!

Fetching the repository.

But we are not done yet. First of all, we need the function that fetches the remote repository.

Because of the infamous git2 library, this part is rather trivial to implement as well:

fn fetch_main_remote(repo: &git2::Repository, config: &Conf) -> Result<()> {
    log::debug!("Fetch: {} / {}", config.origin_remote_name(), config.master_branch_name());
    repo.find_remote(config.origin_remote_name())?
        .fetch(&[config.master_branch_name()], None, None)
        .map_err(Error::from)
}

Here we have a function that takes a reference to the repository as well as a reference to our Conf object. We then, after some logging, find the appropriate remote in our repository and simply call fetch for it. In case of Err(_), we map that to our anyhow::Error type and return it, because the callee should handle that.

Counting the commits

Counting the commits is the last part we need to implement.

fn count_commits_on_main_branch(repo: &git2::Repository, config: &Conf) -> Result<(usize, usize, usize)> {

The function, like the fetch_main_remote function, takes a reference to the repository as well as a reference to the Conf object of our program. It returns, in case of success, a tuple with three elements. I did not add strong typing here, because the codebase is rather small (less than 160 lines overall), so there's not need to be very explicit about the types here.

Just keep in mind that the first of the three values is the number of all commits, the second is the number of merges and the last is the number of non-merges.

That also means:

tuple.0 = tuple.1 + tuple.2

Next, let's have a variable that holds the branch name with the remote, like we're used from git itself (this is later required for git2). Also, we need to calculate the timestamp that is the lowest timestamp we consider. Because our configuration file specifies this in hours rather than seconds, we simply * 60 * 60 here.

    let branchname = format!("{}/{}", config.origin_remote_name(), config.master_branch_name());
    let minimum_time_epoch = chrono::offset::Local::now().timestamp() - (config.hours_to_check() * 60 * 60);

    log::debug!("Branch to count     : {}", branchname);
    log::debug!("Earliest commit time: {:?}", minimum_time_epoch);

Next, we need to instruct git2 to create a Revwalk object for us:

    let revwalk_start = repo
        .find_branch(&branchname, git2::BranchType::Remote)?
        .get()
        .peel_to_commit()?
        .id();

    log::debug!("Starting at: {}", revwalk_start);

    let mut rw = repo.revwalk()?;
    rw.simplify_first_parent()?;
    rw.push(revwalk_start)?;

That can be used to iterate over the history of a branch, starting at a certain commit. But before we can do that, we need to actually find that commit, which is the first part of the above snippet. Then, we create a Revwalk object, configure it to consider only the first parent (because that's what we care about) and push the rev to start walking from it.

The last bit of the function implements the actual counting.

    let mut commits = 0;
    let mut merges = 0;
    let mut nonmerges = 0;

    for rev in rw {
        let rev = rev?;
        let commit = repo.find_commit(rev)?;
        log::trace!("Found commit: {:?}", commit);

        if commit.time().seconds() < minimum_time_epoch {
            log::trace!("Commit too old, stopping iteration");
            break;
        }
        commits += 1;

        let is_merge = commit.parent_ids().count() > 1;
        log::trace!("Merge: {:?}", is_merge);

        if is_merge {
            merges += 1;
        } else {
            nonmerges += 1;
        }
    }

    log::trace!("Ready iterating");
    Ok((commits, merges, nonmerges))
}

This is done the simple way, without making use of the excelent iterator API. First, we create our variables for counting and then, we use the Revwalk object and iterate over it. For each rev, we unwrap it using the ? operator and then ask the repo to give us the corresponding commit. We then check whether the time of the commit is before our minimum time and if it is, we abort the iteration. If it is not, we continnue and count the commit. We then check whether the commit has more than one parent, because that is what makes a commit a merge-commit, and increase the appropriate variable.

Last but not least, we return our findings to the caller.

Conclusion

And this is it! It was a rather nice journey to implement this bot. There isn't too much that can fail here, some calculations might wrap and result in false calculations. Possibly a clippy run would find some things that could be improved, of course (feel free to submit patches).

If you want to run this bot on your own instance and for your own repositories, make sure to check the README file first. Also, feel free to ask questions about this bot and of course, you're welcome to send patches (make sure to --signoff your commits).

And now, enjoy the first post of the bot.

tags: #mastodon #bot #rust

Today, I challenged myself to write a prometheus exporter for MPD in Rust.

Shut up and show me the code!

Here you go and here you go for submitting patches.

The challenge

I recently started monitoring my server with prometheus and grafana. I am no-way a professional user of these pieces of software, but I slowly got everything up and running. I learned about timeseries databases at university, so the basic concept of prometheus was not new to me. Grafana was, though. I then started learning about prometheus exporters and how they are working and managed to setup node exporters for all my devices and imported their metrics into a nice grafana dashboard I downloaded from the official website.

I figured, that writing an exporter would make me understand the whole thing even better. So what would be better than exporting music data to my prometheus and plotting it with grafana? Especially because my nickname online is “musicmatze”, right?

So I started writing a prometheus exporter for MPD. And because my language of choice is Rust, I wrote it in Rust. Rust has good libraries available for everything I needed to do to export basic MPD metrics to prometheus and even a prometheus exporter library exists!

The libraries I decided to use

Note that this article was written using prometheus-mpd-exporter v0.1.0 of the prometheus-mpd-exporter code. The current codebase might differ, but this was the first working implementation.

So, the scope of my idea was set. Of course, I needed a library to talk to my music player daemon. And because async libraries would be better, since I would essentially write a kind of a web-server, it should be async. Thankfully, async_mpd exists.

Next, I needed a prometheus helper library. The examples in this library work with hyper. I was not able to implement my idea with hyper though (because of some weird borrowing error), but thankfully, actix-web worked just fine.

Besides that I used a bunch of convenience libraries:

  • anyhow and thiserror for error handling
  • env_logger and log for logging
  • structopt for CLI parsing
  • getset, parse-display and itertools to be able to write less code

The first implementation

The first implementation took me about four hours to write, because I had to understand the actix-web infrastructure first (and because I tried it with hyper in the first place, which did not work for about three of that four hours).

The boilerplate of the program includes

  • Defining an ApplicationError type for easy passing-around of errors that happen during the runtime of the program
  • Defining an Opt as a commandline interface definition using structopt
#[actix_web::main]
async fn main() -> Result<(), ApplicationError> {
    let _ = env_logger::init();
    log::info!("Starting...");
    let opt = Opt::from_args();

    let prometheus_bind_addr = format!("{}:{}", opt.bind_addr, opt.bind_port);
    let mpd_connect_string = format!("{}:{}", opt.mpd_server_addr, opt.mpd_server_port);

The main() function then sets up the logging and parses the commandline arguments. Thanks to env_logger and structopt, that's easy. The main() function also acts as the actix_web::main function and is async because of that. It also returns a Result<(), ApplicationError>, so I can easily fail during the setup phase of the program.

Next, I needed to setup the connection to MPD and wrap that in a Mutex, so it can be shared between request handlers.

    log::debug!("Connecting to MPD = {}", mpd_connect_string);
    let mpd = async_mpd::MpdClient::new(&*mpd_connect_string)
        .await
        .map(Mutex::new)?;

    let mpd = web::Data::new(mpd);

And then setup the HttpServer instance for actix-web, and run it.

    HttpServer::new(move || {
        App::new()
            .app_data(mpd.clone()) // add shared state
            .wrap(middleware::Logger::default())
            .route("/", web::get().to(index))
            .route("/metrics", web::get().to(metrics))
    })
    .bind(prometheus_bind_addr)?
    .run()
    .await
    .map_err(ApplicationError::from)
} // end of main()

Now comes the fun part, tho. First of all, I have setup the connection to MPD. In the above snippet, I add routes to the HttpServer for a basic index endpoint as well as for the /metrics endpoint prometheus fetches the metrics from.

Lets have a look at the index handler first, to get a basic understanding of how it works:

async fn index(_: web::Data<Mutex<MpdClient>>, _: HttpRequest) -> impl Responder {
    HttpResponse::build(StatusCode::OK)
        .content_type("text/text; charset=utf-8")
        .body(String::from("Running"))
}

This function gets called every time someone accesses the service without specifying an endpoint, for example curl localhost:9123 would result in this function being called.

Here, I can get the web::Data<Mutex<MpdClient>> object instance that actix-web handles for us as well as a HttpRequest object to get information about the request itself. Because I don't need this data here, the variables are not bound (_). I added them to be able to extend this function later on easily.

I return a simple 200 (that's the StatusCode::OK here) with a simple Running body. curling would result in a simple response:

$ curl 127.0.0.1:9123
Running

Now, lets have a look at the /metrics endpoint. First of all, the signature of the function is the same:

async fn metrics(mpd_data: web::Data<Mutex<MpdClient>>, _: HttpRequest) -> impl Responder {
    match metrics_handler(mpd_data).await {
        Ok(text) => {
            HttpResponse::build(StatusCode::OK)
                .content_type("text/text; charset=utf-8")
                .body(text)
        }

        Err(e) => {
            HttpResponse::build(StatusCode::INTERNAL_SERVER_ERROR)
                .content_type("text/text; charset=utf-8")
                .body(format!("{}", e))
        }
    }
}

but here, we bind the mpd client object to mpd_data, because we want to actually use that object. We then call a function metrics_handler() with that object, wait for the result (because that function itself is async, too), and match the result. If the result is Ok(_), we get the result text and return a 200 with the text as the body. If the result is an error, which means that fetching the data from MPD somehow resulted in an error, we return an internal server error (500) and the error message as body of the response.

Now, to the metrics_handler() function, which is where the real work happens.

async fn metrics_handler(mpd_data: web::Data<Mutex<MpdClient>>) -> Result<String, ApplicationError> {
    let mut mpd = mpd_data.lock().unwrap();
    let stats = mpd.stats().await?;

    let instance = String::new(); // TODO

First of all, we extract the actual MpdClient object from the web::Data<Mutex<_>> wrapper. Them, we ask MPD to get some stats() and wait for the result.

After that, we create a variable we don't fill yet, which we later push in the release without solving the “TODO” marker and when we blog about what we did, we feel ashamed about it.

Next, we create Metric objects for each metric we record from MPD and render all of them into one big String object.

    let res = vec![
        Metric::new("mpd_uptime"      , stats.uptime      , "The uptime of mpd", &instance).into_metric()?,
        Metric::new("mpd_playtime"    , stats.playtime    , "The playtime of the current playlist", &instance).into_metric()?,
        Metric::new("mpd_artists"     , stats.artists     , "The number of artists", &instance).into_metric()?,
        Metric::new("mpd_albums"      , stats.albums      , "The number of albums", &instance).into_metric()?,
        Metric::new("mpd_songs"       , stats.songs       , "The number of songs", &instance).into_metric()?,
        Metric::new("mpd_db_playtime" , stats.db_playtime , "The database playtime", &instance).into_metric()?,
        Metric::new("mpd_db_update"   , stats.db_update   , "The updates of the database", &instance).into_metric()?,
    ]
    .into_iter()
    .map(|m| {
        m.render()
    })
    .join("\n");

    log::debug!("res = {}", res);
    Ok(res)
}

Lastly, we return that String object from our handler implementation.

The Metric object implementation my own, we'll focus on that now. It will help a bit with the interface of the prometheus_exporter_base API interface.

But first, I need to explain the Metric type:

pub struct Metric<'a, T: IntoNumMetric> {
    name: &'static str,
    value: T,
    description: &'static str,
    instance: &'a str,
}

The Metric type is a type that holds a name for a metric, its value and some description (and the aforementioned irrelevant instance). But because the metrics we collect can be of different types (for example a 8-bit unsigned integer u8 or a 32-bit unsigned integer u32), I made that type abstract over it. The type of the metric value must implement a IntoNumMetric trait, though. That trait is a simple helper trait:

use num_traits::Num;
pub trait IntoNumMetric {
    type Output: Num + Display + Debug;

    fn into_num_metric(self) -> Self::Output;
}

And I implemented it for std::time::Duration, u8, u32 and i32 – the implementation itself is trivial and I won't show it here.

Now, I was able to implement the Metric::into_metric() function shown above:

impl<'a, T: IntoNumMetric + Debug> Metric<'a, T> {
    // Metric::new() implementation, hidden here

    pub fn into_metric<'b>(self) -> Result<PrometheusMetric<'b>> {
        let instance = PrometheusInstance::new()
            .with_label("instance", self.instance)
            .with_value(self.value.into_num_metric())
            .with_current_timestamp()
            .map_err(Error::from)?;

        let mut m = PrometheusMetric::new(self.name, MetricType::Counter, self.description);
        m.render_and_append_instance(&instance);
        Ok(m)
    }
}

This function is used for converting a Metric object into the appropriate PrometheusMetric object from prometheus_exporter_base.

The implementation is, of course, also generic over the type the Metric object holds. A PrometheusInstance is created, a label “instance” is added (empty, you know why... :–( ). Then, the value is added to that instance using the conversion from the IntoNumMetric trait. The current timestamp is added as well, or an error is returned if that fails.

Last but not least, a new PrometheusMetric object is created with the appropriate name and description, and the instance is rendered to it.

And that's it!

Deploying

The code is there now. But of course, I still needed to deploy this to my hosts and make it available in my prometheus and grafana instances.

Because I use NixOS, I wrote a nix package definition and a nix service defintion for it, making the endpoint available to my prometheus instance via my wireguard network.

After that, I was able to add queries to my grafana instance, for example:

mpd_db_playtime / 60 / 60 / 24

to display the DB playtime of an instance of my MPD in days.

I'm not yet very proficient in grafana and the query language, and also the service implementation is rather minimal, so there cannot be that much metrics yet.

Either way, it works!

A basic dashboard for MPD stats

Next steps and closing words

The next steps are quite simple. First of all, I want to make more stats available to prometheus. Right now, only the basic statistics of the database are exported.

The async_mpd crate makes a lot of other status information available.

Also, I want to get better with grafana queries and make some nice-looking graphs for my dashboard.

Either way, that challenge took me longer than I anticipated in the first place (“I can hack this in 15 minutes” – famous last words)! But it was fun nonetheless!

The outcome of this little journey is on crates.io and I will also submit a PR to nixpkgs to make it available there, too.

If you want to contribute patches to the sourcecode, which I encourage you to do, feel free to send me patches!

tags: #prometheus #grafana #rust #mpd #music

Happy new year!

I just managed to implement syncthing monitoring for my prometheus and grafana instance, so I figured to write a short blog post about it.

Note: This post is written for prometheus-json-exporter pre-0.1.0 and the configuration file format changed since.

Now, as you've read in the note above, I managed to do this using the prometheus-json-exporter. Syncthing has a status page that can be accessed with

$ curl localhost:22070/status

if enabled. This can then be used to push to prometheus using the prometheus-json-exporter mentioned above using the following configuration for mapping the values from the JSON to prometheus:

- name: syncthing_buildDate
  path: $.buildDate
  help: Value of buildDate

- name: syncthing_buildHost
  path: $.buildHost
  help: Value of buildHost

- name: syncthing_buildUser
  path: $.buildUser
  help: Value of buildUser

- name: syncthing_bytesProxied
  path: $.bytesProxied
  help: Value of bytesProxied

- name: syncthing_goArch
  path: $.goArch
  help: Value of goArch

- name: syncthing_goMaxProcs
  path: $.goMaxProcs
  help: Value of goMaxProcs

- name: syncthing_goNumRoutine
  path: $.goNumRoutine
  help: Value of goNumRoutine

- name: syncthing_goOS
  path: $.goOS
  help: Value of goOS

- name: syncthing_goVersion
  path: $.goVersion
  help: Value of goVersion

- name: syncthing_kbps10s1m5m15m30m60m
  path: $.kbps10s1m5m15m30m60m
  help: Value of kbps10s1m5m15m30m60m
  type: object
  values:
    time_10_sec: $[0]
    time_1_min: $[1]
    time_5_min: $[2]
    time_15_min: $[3]
    time_30_min: $[4]
    time_60_min: $[5]

- name: syncthing_numActiveSessions
  path: $.numActiveSessions
  help: Value of numActiveSessions

- name: syncthing_numConnections
  path: $.numConnections
  help: Value of numConnections

- name: syncthing_numPendingSessionKeys
  path: $.numPendingSessionKeys
  help: Value of numPendingSessionKeys

- name: syncthing_numProxies
  path: $.numProxies
  help: Value of numProxies

- name: syncthing_globalrate
  path: $.options.global-rate
  help: Value of options.global-rate

- name: syncthing_messagetimeout
  path: $.options.message-timeout
  help: Value of options.message-timeout

- name: syncthing_networktimeout
  path: $.options.network-timeout
  help: Value of options.network-timeout

- name: syncthing_persessionrate
  path: $.options.per-session-rate
  help: Value of options.session-rate

- name: syncthing_pinginterval
  path: $.options.ping-interval
  help: Value of options.ping-interval

- name: syncthing_startTime
  path: $.startTime
  help: Value of startTime

- name: syncthing_uptimeSeconds
  path: $.uptimeSeconds
  help: Value of uptimeSeconds

- name: syncthing_version
  path: $.version
  help: Value of version

When configured properly, one is then able to draw graphs using the syncthing-exported data in grafana.

There's nothing more to it.

tags: #nixos #grafana #prometheus #syncthing

My between-the-years project was trying to use my old Thinkpad to run some local services, for example MPD. I thought, that the Thinkpad did not even have a drive anymore, and was surprised to find a 256GB SSD inside of it – with nixos still installed!

AND RUNNING!

So, after almost two years, I booted my old Thinkpad, entered the crypto password for the harddrive, and got greeted with a login screen and an i3 instance. Firefox asked whether I want to start the old session again... everything just worked.

I was amazed.

Well, this is not the crazy thing I wanted to write about here. The problem now was: I update and deploy my devices using krops nowadays. This old installation had root login disabled, which is required for krops to work...

But, because nixos is awesome,... I did nothing more than checking out the git commit the latest generation on the Thinkpad was booted from, modified some settings for ssh server and root user ... rebuild the system and switched to the new build... and then started deployment for the new nixos 20.09 installation using krops.

All without hassle. I might run out of disk space now, because this deployes a full KDE Plasma 5 installation, but honestly it would surprise me... there should be enough space. I am curious, though, whether KDE Plasma 5 runs on the device. We'll see...

tags: #nixos #desktop

I consider myself a power user. I've been using Linux exclusively on all my machines (except maybe on my Androids, if you do not consider these linuxes) for over ten years. Most of the time, I used i3wm (and some time sway), but I got tired of it about two years ago. I switched to XFCE. I told myself, I'm living inside a terminal anyways, I use mutt for mails, (neo)vim for editing stuff and do not use that many GUI apps anyways. The browser, of course. And telegram-desktop. Recently (about a year ago maybe?), riot-desktop joined the two.

I told myself that XFCE is eye-candy enough for me and performs really nice without getting in my way while working in terminals.

Today, I switched my two main devices to KDE Plasma 5. Well, my notebook was actually switched yesterday. And because it was such a huge leap in “niceness”, I switched my workstation today.

I also switched my email setup to Korganizer with Kmail for mails. I am looking forward to see how that performs for me.

I always considered KDE the desktop for Linux. The one, normal users should use to get a first impression of Linux. The one, that will be there for “Linux on the Desktop” when it gets real. You know what I mean?

I never considered for a power-user setup. Power users use i3wm, awesomeWM, sway, ... but not full-blown desktop environments, do they? Well, on several occasions I met Kai Uwe, a KDE dev from a town not far from the town I grew up in. We've met on Linux-Days and other Community events. I always was amazed how fast KDE was on his machines. I was blown away by the usability. But I never really considered switching.

Until today.

Memory usage is where it should be: My desktop uses 3.6GB right now, with Cantata (MPD client), Riot-Desktop (Electron!), Telegram, Firefox and Kontact open (plus some more terminals and some services running in the background as well).

My Notebook was at about 5GB RAM usage with the same apps open and even some more stuff running in the background.

After that experience and also that performance on my rather old desktop machine, I'm blown away and am willing to invest time into getting to know how to be even faster and more productive than I am currently feeling with KDE.

tags: #kde #linux #desktop #software

Holy crap, I haven't written on my blog for a long time. And I almost missed that the Rust community asked for blog posts about Rust in 2021 - but I am in time I guess, so here it goes.

Most Rustaceans won't agree with this blog post, I guess. But I also think that's fine, because that's the whole point of the Blog-Post-For-The-Roadmap thing, right? Asking people for different opinions and starting a constructive discussion about the topic. I also must say that I haven't read a single one of the other Rust-2021 Blog posts just yet.

I also think this will be rather short, but I hope I express my feelings in the best way possible for you all to understand.

Don't Change!

I got into Rust at about Rust 1.5.0. After the first half of 2020, I felt like January was years ago, so I feel like Rust 1.5.0 was in another lifetime. So much happened this year, and still, so little was accomplished by me and my friends. The world turned upside down, essentially.

Rust changed a lot between 1.5.0 and the current compiler I have installed on my system:

$ rustc --version
rustc 1.46.0 (04488afe3 2020-08-24)

The RELEASES.md file is a whooping 9167 lines long. We got cargo workspaces, we got awesome things like the ? operator (which I definitively was not a friend of in the beginning), we got associated constants, incremental compilation, impl Trait, we got const functions and most importantly we got async/await.

Fairly, that's where I started to struggle to keep up. I definitively see the value in async-await and what it actually enables us to do with Rust, and how to do it. But I just couldn't keep up with the change anymore. It was too much. I couldn't cope learning all these new things just in time they arrived. I, to this day, struggle to write a simple Program with async/await if there's too much iterators involved. I don't know where my actual problems are, because I cannot see through the whole concept enough to understand what I am doing wrong.

The last five years were full of change. Good change, of course. But this last (almost)year just drowned me. Too much to handle.

My hope is, that Rust does not change anymore when it comes to features. I see that there is a lot of demand for const generics, especially by the embedded community. I understand why. I hope it doesn't have any impact on me as a commandline-program-writing Rustacean.

But...

But. There's always a “but”, isn't there?

I still have high hopes for some things concerning Rust. But they do not at all have to do with the language Rust, but the environment around it. As stated before, I'm a commandline-program writing person. I do not write web services (yet?), I do not write embedded stuff, I do not write high-performance/performance-critical stuff.

Essentially: I write programs in Rust that others would write in Python, Ruby or Node. I write them in Rust because I am a lazy programmer, because I do not care enough. I write Rust, because the compiler YELLS at me to get it right. If I would do the same thing in Ruby, my go-to-language for everything below 100LOC, I would get myself into a wheelchair because I would use every footgun available.

Deep inside, I'm a bad programmer and rustc forces me to be a good one.

That was a bit of a rant, I hope you're still with me. The paragraph above was for you to understand where I come from. I don't care if my program runs in 1 second or 10, because the domain I write for does not care most of the time. But what is important to me, is that I actually can write my programs. Often, I cannot. And that's simply because of one thing:

Libraries are missing.

Those who know me knew that this was coming. Libraries for domains that I care about are missing. That is calendar (icalendar) reading/writing, vcard reading/writing, email reading/writing (the format, not the networking stuff), ... There are already libraries out there for these things, although they are far from being complete, usable or even correct. Writing a simple TUI MUA for notmuch is pain right now, because parsing email is really hard in Rust, and there are no high-level libraries available. The “mail” crate ecosystem is closest, but they do not yet have a parser.

There are, I am sure, more things in this part of the ecosystem (that is libraries for basic formats) where Rust could shine, but does not yet.

My request for Rust in 2021 is: Make things shiny. Make them available, make them work, make them correct, make them nice to use (E.G. parsing mails into tokens and handing them to me is okay, but having a high-level interface is much nicer).


To sum it up in one sentence:

Don't change rust itself, but improve the library ecosystem.

That's my hope for Rust 2021. Thank you for having me in this awesome community and thank you for reading.

tags: #rust #programming

Because I want to use notmuch for my mails and also sync tags between my machines, I figured I should install notmuch and muchsync on my uberspace. But because centos7 does not have packages and uberspace does not yet provide a way to conveniently install packages for a user, I had to compile the world.

Here's the guide for doing it.

Preface

In the following steps, I will download, unpack and build sources from scratch. My download and compile folder will be at $HOME/compile. My install folder will be at $HOME/install. I don't be installing stuff into my $HOME directly on the uberspace, because I want to be a bit organized here.

Thus, you might want to adapt your commandline calls.

Obtaining sources

First, you should go and fetch all latest sources for the whole dependency chain. I will not link to the individual tarballs, you should figure out the links for yourself to get the latest and greatest of each project.

Here's the list of links where you'll find the code you'll need to compile:

Configuring the build(s) and building

First of all, execute

mkdir -p $HOME/install/lib/pkgconfig
export PKG_CONFIG_PATH=$HOME/install/lib/pkgconfig/:$PKG_CONFIG_PATH

To make sure the pkg-config utility finds the right package configuration files.

It follows a list of configure calls for the respectice library/binary. I hope this list is complete and I didn't miss anything in the process. Feel free to mail-ping me if something is missing.

For all packages that are not listed here, a simple

./configure --prefix=$HOME/install && make install

did the job. For the following packages, it was not as convenient, but after successfully configuring, you can install all of them with a simple make install.

libassuan

./configure --prefix=$HOME/install \
    --with-libgpg-error-prefix=$HOME/install/

gpgme

./configure --prefix=$HOME/install \
    --with-libgpg-error-prefix=$HOME/install \
    --with-libassuan-prefix=$HOME/install

libksba

./configure --prefix=$HOME/install \
    --with-libgpg-error-prefix=$HOME/install

gmime

PATH=$HOME/install/bin:$PATH ./configure --prefix=$HOME/install

libgcrypt

./configure --prefix=$HOME/install \
    --with-libgpg-error-prefix=$HOME/install

ntbtls

./configure --prefix=$HOME/install \
    --with-libgpg-error-prefix=$HOME/install \
    --with-libgcrypt-prefix=$HOME/install \
    --with-ksba-prefix=$HOME/install

gnupg

./configure --prefix=$HOME/install \
    --with-libgpg-error-prefix=$HOME/install \
    --with-libgcrypt-prefix=$HOME/install \
    --with-libassuan-prefix=$HOME/install \
    --with-ksba-prefix=$HOME/install

notmuch

You might want to open a new shell for the environment variables

export LD_LIBRARY_PATH=$HOME/install/lib:$LD_LIBRARY_PATH PATH=$HOME/install/bin:$PATH
export LDFLAGS="$LDFLAGS -L$HOME/install/lib/ "
export CFLAGS="$CFLAGS -I$HOME/install/include/ -std=c99 "
export PKG_CONFIG_PATH=$HOME/install/lib/pkgconfig:$PKG_CONFIG_PATH

./configure --prefix=$HOME/install \
    --without-bash-completion \
    --without-docs \
    --without-api-docs \
    --without-emacs \
    --without-desktop \
    --without-ruby \
    --without-zsh-completion

muchsync

You might want to open a new shell for the environment variables

export LDFLAGS="$LDFLAGS -L$HOME/install/lib/ -Wl,-rpath,$HOME/install/lib/"
export LD_LIBRARY_PATH=$HOME/install/lib:$LD_LIBRARY_PATH
export CXXFLAGS="$CXXFLAGS -I$HOME/install/include "
export PKG_CONFIG_PATH=$HOME/install/lib/pkgconfig

./configure --prefix=$HOME/install --exec-prefix=$HOME/install

List of versions this guide was written for

This is the (alphabetically sorted) list of stuff I had to compile and install, including versions so you can figure whether you have something newer and might need to investigate on differences when compiling your stuff:

  • gmime-3.2.7
  • gnupg-2.2.21
  • gpgme-1.14.0
  • libassuan-2.5.3
  • libgcrypt-1.8.6
  • libgpg-error-1.38
  • libksba-1.4.0
  • muchsync-5
  • notmuch-0.30
  • npth-1.6
  • ntbtls-0.1.2
  • talloc-2.3.1
  • xapian-core-1.4.16

Of course you should update your installation if new versions come out.

Happy compiling.

tags: #uberspace #email #linux

Finally, I managed to implement a proof of concept of serde-select. But lets start at the beginning.

The Problem

The problem I tried to solve with this crate is rather simple: You need to be able to get values from a serde-compatible document (e.g. toml, json, yaml, ...) but you don't know the full schema of the document at compiletime of your crate.

The origin of the idea of serde-select was when I first started working on my imag project, where a lot of seperated crates coexist in one ecosystem, but all of them should be configured in one big configuration file. Of course I did not want to have one central crate just for defining the schema, especially since a user might not want to use all functionality from the ecosystem, thus not having a “full” configuration file, but only the parts they needed.

So I started writing “toml-query”, a crate which lets the programmer query a toml::Value with a “path”. For example:

[calendar]
list_format = "{{lpad 5 i}} | {{abbrev 5 uid}} | {{summary}} | {{location}}"
show_format = """
{{i}} - {{uid}}
"""

[ref]
[ref.basepathes]
music = "/home/user/music"
contacts = "/home/user/contacts"
calendars = "/home/user/calendars"

The document looks like this, but in the program code we only need calendar and its sub-values. So we can do

let r = document.read("calendar.list_format");

in the code and get a Result<Option<&'document Value>> value back.

toml-query evolved over the time, now featuring more flexibility by implementing “Partials”, how I call them. These are structs that are Serialize + Deserialize and have a path attached to them, so deserializing the partial document is possible right away:

let r: Result<Option<CalendarConfig>, _> = document.read_partial::<CalendarConfig>();

where CalendarConfig: Serialize + Deserialize + Debug + toml_query::Partial (see here).

The evolution

toml-query works perfectly fine and I use it in my other projects a lot. It is fast and easy to use. Its error reporting is nice.

But an idea formed in the back of my head and I did not stop to think about it.

Can toml-query by generalized to work with all formats serde can handle?

So I started to experiement with a more general implementation: serde-select was born.

And today I managed to get the first bits working.

Meet serde-select

serde-select implements a “read” functionality for both JSON and TOML, depending on what features you enable. The inner implementation of the resolve-algorithm is agnostic of the actual format.

For a quick overview how to use the crate right now, have a look at the tests for toml for example.

I strongly advice against using this crate, though. It is only an experiement for now and shouldn't be used in production code. Nevertheless, I published the first preview on crates.io.

tags: #rust #programming

In the last few months, I was invited to join the nixos organization on github multiple times. I always rejected. Here's why.

Please notice

Please notice that I really try to write this down as constructive criticism. If any of this offends you in any way, please let me know and I'll rephrase the specific part of this article. I really do care about the nixos community, I've been a user of NixOS (on all my devices except phone) since mid 2014, I've been a contributor since January 2015 and I am continuing to be an user and an author of contributions.

I do think that Nix or even NixOS is the one true way how to deploy systems that need to be reproducible, even if that needs one to sacrifice certain comfort.

Context

Secondly, I need to provide some context from where I'm coming so the dear reader can understand my point of view in this article.

First of all, I did not start my journey with NixOS, of course. I was a late bloomer in regards to linux, in fact. I was introduced to Ubuntu by a friend of mine in 11th grade. I started to use Kubuntu, but only a few weeks later my friend noticed that I was getting better and better with the terminal, so maybe not even half a year later I switched to Archlinux, which I used on my desktops until I was introduced to NixOS. In that time, I learned how to write Java (which I do not do anymore btw), Ruby and C, started hacking a lot of funny things and managed to contribute patches to the linux kernel about two years later.

I'm not trying to show balls here! That last bit is important for this article, especially if you know how the kernel community works and how the development process of the kernel works. I guess you know where this is going.

I heard of NixOS in late 2014 at a conference in the black forest, where Joachim Schiele talked about it. A few months later, my latex setup broke from an update and I was frustrated enough by Archlinux to try something new.

I never looked back.

The “early days”

When I started using NixOS, Nix, the package manager, already existed for about ten years. Still, the community was small. When I went on the IRC channel or on the mailinglist, I could easily remember the nicknames and I was able to skim through the subjects of the mails on the list to see what was going on, eventhough I did not understand all of it.

That soon changed. I remember the 15.09 release when everyone was super excited and we were all “yeah, now we're beginning to fly” and so on. Fun times!

Problem 1: Commit access and development process

Now, lets get into the problems I have with the community and why I reject the invitations to join the github organization.

The problem

In fact, I started people asking and telling about this pretty early on: five(!) years ago, I started replying to an email thread with this message

Quote:

Generally, I think it would be best to prevent commit access as far as possible. Having more people to be able to commit to master results in many different opinions committing to master, which therefor results in discussions, eventually flamewars and everything.

Keeping commit access for only a few people does not mean that things get slower, no way!

[...]

What you maybe want, at least from my point of view, is staging branches. Some kind of a hierarchy of maintainers, as you have in the linux kernel. I fully understand that the linux kernel is a way more complex system as nixos/nixpkgs, no discussion here. But if you'd split up responsibilities, you may end up with

* A fast and secure development model, as people don't revert back and forth.

* Fewer “wars” because people disagree on things

* Less maintaining efforts, because the effort is basically split up in several small “problems”, which are faster to solve.

What I want to say is, basically, you want a well-defined and structured way of how to do things.

Also please note that there's another mail from Michael Raskin in that thread where we talked about 25 PRs for new packages. Right now we're at about 1.8k open pull requests, with over 580 of them for new packages.

I take that as proof that we did not manage to sharpen and improve the process.

Lets get to the point. I started telling people that the development process we had back then was not optimal. In fact, I saw it coming: The community started to grow at an great pace back then and soon I talked to people on IRC and Mailinglist where I was like “Who the hell is this, I've never seen this name before and they seem not to be new, because they already know how things work and teach me...“.

The community grew and grew, over 4500 stars on github (if that measures anything), over 4500 forks on github.

When we reached 1k open pull requests, some people started noticing that we might not be able to scale anymore at some point. “How could we possibly manage that amount of pull requests ever?“.

Now we're at about 1.8k open pull requests.

I proposed changes several times, including moving away from github, which does IMO not scale to that amount of issues and PRs, especially because of its centralized structure and because of its linear discussions.

I proposed switching to kernel-style mailinglist. I was rejected with “We do not have enough people for that kind of development model”. I suspect that people did not understand what I meant by “kernel-style” back then (nor do I think they understand now). But I'm sure, now more than ever, that a switch to a mailinglist-based development model, with enough automation in place for CI and static analysis of patches would have had the best possible impact for the community. Even if that'd mean that newcomers would be a bit thrown-off at first!

The current state of affairs is even worse. Right now (as of this commit) , we have

  • 1541 merges on master since 2020-01-01
  • 1601 patches pushed directly to master since 2020-01-01

Feel free to reproduce these numbers with

$ git log --oneline --first-parent --since 2020-01-01 --[no-]merges | wc -l

That means that we had 1601 possibly breaking patches pushed by someone who things they are good enough and that their code never breaks. I'll leave it to the dear reader to google why pushing to master is a bad idea in a more-than-one-person-project.

Another thing that sticks out to me is this:

$ git log  --first-parent --since 2020-01-01 --merges | \
    grep "^Author" |    \
    sort -u |           \
    wc -l
74

74! 74 people have access to the master branch and can break it. I do not allege incompetence to any of these people, but we all know that not always everything works as smoothly as we expected, especially in software development. People are tired sometimes, people do make mistakes, people do miss things when reviewing things. That's why we invented continuous integration in the first place! That some thing can check whether the human part of the process did the right thing and report back if they didn't.

The solution

My dream-scenario would be that nobody would have access to master except for a bot like bors (or something equivalent for the Nix communiy). The rust communit, which uses bors heavily does software develoment the right way. If all checks pass, merging is done automatically. If not, the bot finds the breaking change by using a clever bisecting algorithm and merges all other (non-breaking) changes.

In fact, I would go further and introduce teams. Each team would be responsible for one task in the community. For example there's different packaging ecosystems within the nixpkgs repository, one for every language. Each language could get a team of 3 to 5 members that coordinate the patches that come in (from normal contributors) and apply them to a <language>-staging branch. That branch would be merged on a regular basis (like... every week) to master, if all tests/builds succeed (just like the kernel community does it)!

A team could also be introduced for some subsets of packages... Qt packages, server software, but also nixpkgs-libs or even documentation (which is another subject on itself).

Problem 2: “Kill the Wiki”

In 2015, at the nixcon in Berlin, we had this moment with “Kill the Wiki”. As far as I remember it was Rok who said that (not sure though). I was not a fan back then, and I'm actually even less a fan of that decision now.

Killing the wiki was the worst thing we could do documentation-wise. Everytime I tell people about nixos, I have to tell them that there is no decent documentation around. There is, of course, the documentation that is generated from the repository. That one is okay for the initial setup, but it is more than far away from being a good resource if you just want to look up how some things are done.

The nixos.wiki efforts fill the gap here a bit, sure. But we could really do better.

The solution would be rather simple: Bring back a wiki software, even if we start from scratch here or “just” merge the efforts from nixos.wiki – or make that one the official one – it would be an improvement all the way!

Problem 3: “Kill the mailinglist”

Certainly, what does this community have with killing their own infrastructure? They killed the wiki, they killed the mailinglist... both things that are really valuable... but github is the one thing that actually slows us down ... and does not get killed... I am stunned, really.

The solution here is also really simple: Bring it back. And not googlegroups or some other shitty provider, just host a mailman and create a few mailinglists... like the kernel.

I hope I do not have to write down the benefits here because the reader should be aware of them already. But for short:

  • Threaded discussions (I can reply multiple times to one message, quote parts and reply to each part individually, creating a tree-style discussion where each branch focuses on one point)
  • Asyncronous discussions (I can reply to a message in the middle of a thread rather than appending)
  • Possibility to work offline (yeah, even in our age this is important)
  • User can choose their interface (I like to use mutt, even on my mobile if possible. Web UIs suck)

I am aware that the “replacement” (which it really isn't) discourd is capable of going into mailinglist-mode. Ask me how great that is compared to a real mailinglist!

It is not.

The silver lining...

This article is a rather negative one, I know that. I do not like to close words with that negative feeling.

In fact, we got the RFC process, which we did not have when I started using nixos. We have the Borg bot, which helps a bit and is a great effort. So, we're in the process of improving things.

I'm still positive that, at some point, we improve the rate of improvements as well and get to a point where we can scale up to the numbers of contributors we currently have, or even more.

Because right now, we can't.

Errata

I did make some mistakes here and I want to thank everyone for telling me.

Numbers

Some nice folks on the nixos IRC/matrix channel suggested that my numbers for PRs vs. pushes to master were wrong, as githubs squash-and-merge feature is enabled on the github repository for nixpkgs.

It seems that about 4700 PRs were merged since 2020-01-01. This does proof my numbers wrong. Fact is: on my master branch of the nixpkgs github repository, there are 3142 commits. It seems that not all pull-requests were to master, which is of course true because PRs can and are filed against the staging branch of nixpkgs and also the stable branches.

Github does not offer a way to query PRs that are filed against a certain branch (at least not in the web UI), as far as I see.

So let's do some more fine-granular analysis on the commits I see on master:

git log --oneline --first-parent --since 2020-01-01 | \
    grep -v "Merge pull request" | \
    wc -l
1650

As github does create a commit message for the merge, we can grep that away and see what the result is. I am assuming here that nobody ever changes the default merge commit message, which might not be entirely true. I assume, though, that it happens not that often.

So we have 3142 commits from which are 1650 not github-branch-merges.

From time to time, master gets merged into staging and the other way round:

  • 20 merges from master to staging
  • 5 merges from staging to master

That leaves us at 1625 commits where the patch landed directly on master. How many of these patches were submitted via a pull request is not that easy to evaluate. One could write a crawler that finds the patches on github and checks whether they appear in a PR... but honestly, my point still holds true: If only one breaking patch lands on master per week, that results in enough slow-down and pain for the development process.

The inconsistency in the process is the real problem, having a mechanism that handles and schedules CI jobs and merges and a clear merge-window for per-topic changesets from team-maintained branches would give the community some structure. New contributors could be guided more easily as they would have a counterpart to contact for topic-specific questions and negotiations wouldn't be between people anymore but between teams, which would also give the whole community some structure and would also clearify responsibilities.

tags: #nixos #community

The call for blogs was just issues a few days ago – and here I am writing about my biggest pains this year... because that's what the call for blogs basically is for me... I write down my pains with Rust and hope things get better slowly next year.

Don't misunderstand what I want to say here though: Rust is awesome, has an awesome community, awesome tooling, awesome everything... well not completely (because otherwise I wouldn't have to write this article, right?), but almost.

Pain #0: Libraries

Pain Number Zero (because that's where computer programmers start to count) is the library ecosystem. “What?” you say? “Rust is known for a really good library ecosystem although the language is not even five years old (counting from 1.0.0)“, you might say! And of course you're right... but not in my domain, unfortunately.

Rust has excellent libraries for developing web services backend as well as backend, gaming engines and games and of course microcontroller stuff. But these are not my domains. My domain is commandline user stuff. My domain might become TUI applications or even GUI applications in the future. My domain is data formats, especially icalendar and vcard, because I write journal applications, calendar applications, contact management stuff, todo applications and diary tooling and even Email processing/handling stuff and possibly even a CLI/TUI mail reader – of course I'm talking about imag here, the text-based, commandline personal information management suite I'm developing over four years now.

The tooling in this domain is not nonexistent, no way! But, despite the efforts some people in this awesome community started, the number and especially the quality of those libraries is nowhere as satisfying as the support for other domains. No offense to the libraries authors of course! It is not their fault at all. It is just that only a few people have started initiatives in this direction yet. I try to contribute! I am actually working on a libical high-level frontend which I started to extract from khaleesi, a work by two wonderful people which includes a wrapper around libical and libical-sys that I started to extract into a library crate.

But there could be so much more and better support for these things! I can only do so much. So I call out: Help developing libraries for these standards! Especially help developing high-level Rust libraries for these things, because handling mime as a way to work with emails is just the beginning. Parsing mail into something that can be worked with on a high level in Rust would be a wonderful goal for 2020. And of course all the other domains!

I remember from my days with Ruby that code could be written at a high level when working with mail and other such formats. Lets have that in Rust!

Lets have world-class support for handling data formats at a high level, so we can write “Speaking code” like it would be plaintext.

Pain #1 – CLI

My next pain are frontends. What I mean by that is CLI, TUI and GUI frontends, not WUI (web user interface) frontends. But I'll break this down into several sections here, so let's talk about CLI first...

One thing here is of course the wonderful clap crate. Lets make clap v3.0 happen next year! It would be a huge step forward!

But this is only one minor pain point, because clap is already a wonderful thing. Lets also make the interactive commandline user interface story better!

I remember that the people from the Node community have commandline applications that you can use interactively that just amaze me because they are so comfy to use (and I'm not even talking about TUI applications here, just interactive CLI apps)! I think we can have this in Rust!

Lets have the best libraries to implement interactive commandline applications!

Pain #2 – TUI

TUI is the next thing i want to point out. Short disclaimer though: I never wrote a TUI app, but I certainly plan to do so. Maybe not in 2020 but after that, imag should get a TUI interface at some point. And for that, of course, I would love to have a headache after thinking about which library or framework to chose.

Right now, there's cursive – and holy swearword this thing looks amazing! But there could be so much more, still! Of course there are already a few extension crates out there:

(btw: @deinstapel you're a hero – they implemented half of the crates above!)

But I bet there could be more... I could think of an embedded terminal for cursive, I can think of an editor-view for embedding vim or another TUI editor into a cursive application, I could think of an editor-like view embedding the Xi editor...

Lets make Rust the go-to choice for writing TUI applications!

Pain #3 – GUI

And of course, the GUI domain. I could write up a long text here, or just point you to other Rust-2020 articles that expand on the topic... but I don't. Why? Because I never implemented a GUI application, I don't see myself implementing one in the near (or even far) future (at least not for imag) and so I don't take the liberty to reiterate what others said more eloquently: The Rust ecosystem for writing GUI applications is not good.

Lets improve our GUI-writing experience!

Summary

All in all, I hope that 2020 will be the year of the Rust Language as application language. We have an awesome tooling and frameworks available for web stuff, the game-domain is expanding constantly and low-level programming is possible and done out there all the time.

Writing applications in Rust is not yet as awesome as it could be, though. So my hopes, dreams and wishes are...

Lets make Rusts high-level application writing experience the best out there!

tags: #rust