musicmatzes blog

rust


“Thoughts” is (will be) a weekly roll-up of my mastodon feed with some notable thoughts collected into a long-form blog post. “Long form” is relative here, as I will only expand a little on some selected subjects, not write screens and screens of text on each of these subjects.

If you think I toot too much to follow, this is an alternative to follow some of my thoughts.


This week (2021-05-29 – 2021-06-04) I got very angry about this “you need an app for this” bullshit and some things died.

App madness

I am very angry (german) about every other service forcing me to install an app of some sort or another on my devices. This time it was my insurance that wanted me to install either a desktop application (Windows and Mac only) or a Smartphone app (Android or iOS only) just to update my bank account or address. How mad can they possibly be?

I reported them to #digitalcourage.

github actions

I probably said it before, but the more I play with #githubactions, the more I like it.

This is mainly due to the fact that the features are well-designed. You can make dependend jobs and you can even boot up #docker containers, if your application has to be tested against, for example, a database or some other service. I know that #github will never #opensource this, that's why I hope someone implements it as a FLOSS alternative!

Awesome Rust

It always amazes me how good the #rust standard library actually is. I was able to solve an issue in my codebase with a two-line patch that would have been way more complex in another language!

Dying things

First audacity, now stackoverflow (german) died.

What's next? I really hope we can develop alternative FLOSS platforms. For audacity, there are several alternative tools around that one can switch to. For stackoverflow, not so much. Especially because the software is only one part, the other part is the data. There are dumps of stackoverflow somewhere on the internet (I'm not linking because I don't know how legal these dumps actually are), so maybe someone can implement a FLOSS alternative (please make it federated or distributed) and import that dataset?

This would be awesome!

Recently, I voiced my discomfort... no, lets face it: my anger with people that cannot obey the simplest commit message rules:

Why can't people obey these simple #commit message rules?

  • Use an uppercase letter for the start of your subject line
  • EXPLAIN what you did, not “Fixes”

#git

(toot)

This really bothers me. I (co-)maintain a few crates in the #rust ecosystem. There are contributions rolling in every other week and I love that, because it makes me happy to see that other people care about the same things that I care about. Still, I am constantly asking people to rewrite commit messages or clean up their branches because they did strange things – for example merge the master branch instead of rebasing their pull-request to fix merge conflicts. And sometimes even change things in this merge commit, making a review utterly impossible!

Most of the time I do not bother if people just don't capitalize the first letter of their commit message, but it bothers me to no end, still. That's why I teach others to write proper commit messages when I teach them how to use #git, and I really try to be a pain in their ... youknowwhat, so they are annoyed by me telling them “No, rephrase that!” all the time!

I am not angry if people fail to use git trailers the right way (and yes, these are kernel commit conventions. Does not mean they cannot be applied to other workflows as well)! These rules are, of course, not carved into stone. Still, it is a matter of good behaviour in the community to give attribution to people involved in the process of applying the patch (using “Signed-off-by”, “Acked-by” or “Reviewed-by”), crafting the patch (using “Co-authored-by”, “Suggested-by”, “Signed-off-by”) or others (“CC”, “Reported-by”, ...).

I hope I don't have to repeat that commit messages like “Fixes” or “Refactor” are bullshit!

How to NOT do better

There are projects out there that try to make you a better committer. Most known is conventional commits.

I don't like these things at all. “Why?” you may ask? The answer to that is really simple: It makes you think less about what you've done, and, and that's propably the worst thing, it gives you the ability to auto-generate a changelog from your commit logs. But commit logs are not changelogs. Commit logs are logs of steps how your software was developed. A changelog is a list of things your users need to know about when upgrading from one version to another. They don't need to know the steps that where taken to provide new features, fix bugs or refactor your codebase, they need to know about what changes for them, how using the product has changed!

Luckily I have managed to stay away from projects using conventional commits.

How to do better

There are tons and tons of guides out there how to write proper git commit messages. I leave searching for them as a task for the reader here (one thing I want to link here is Drew DeVault's article on a disciplined git workflow). The very basics are:

  • The subject line must not exceed 50 characters
  • The subject line should be capitalized and must not end in a period (really, who on earth would end it with a period? I mean... do you end your email subjects with a period?)
  • The subject line must be written in imperative mood (“Fix”, not “Fixed” / “Fixes” etc.)
  • The body copy must be wrapped at 72 columns
  • The body copy must only contain explanations as to what and why, never how. The latter belongs in documentation and implementation.

There are, of course tons of great examples out there. And because people get annoyed if I tell them that the best examples can be found in the linux kernel community (“These people are GODs, I cannot compare to them” – why not?), I can only show you some less GODish commits (by me)!

Have a look at my

  • contribution to shiplift. The commit message is not that long and the change is atomic. The subject line is a short summary what the change is about, the body explains why this is/was done. The trailer notes that I submitted this according to the developercertificate.
  • contribution to config-rs. Nobody said that the commit subject has to explain all the things – as long as there is reasonable explanation in the body, the subject can be “Simplify implementation” (like here).
  • commit bringing order to the galaxy. There can be the occasional joke, of course!

These are all rather short commit messages for simple patches. Longer messages with more explanations also exist in my projects! For example in the butido project there are changes like this one, or this one or even this very long one. Or, to go crazy, this enormous one here.

These commits have one thing in common: They explain why things were done.

And you can do that to! One really simple idea that is worth trying out is not to use the -m flag of git-commit at all. This way you are presented with your favourite editor and can pause for a moment to think about what to write.

Don't be that guy that appears on the front page of commitlogsfromlastnight.com!

Today, I wrote a mastodon bot.

Shut up and show me the code!

Here you go.

The idea

My idea was, to write a bot that fetches the lastest master from a git repository and counts some commits and then posts a message to mastodon about what it counted.

Because I always complain about people pushing to the master branch of a big community git repository directly, I decided that this would be a perfect fit.

(Whether pushing to master directly is okay and when it is not okay to do this is another topic and I won't discuss this here)

The dependencies

Well, because I didn't want to implement everything myself, I started pulling in some dependencies:

  • log and env_logger for logging
  • structopt, toml, serde and config for argument parsing and config reading
  • anyhow because I don't care too much about error handling. It just has to work
  • getset for a bit cleaner code (not strictly necessary, tbh)
  • handlebars for templating the status message that will be posted
  • elefren as mastodon API crate
  • git2 for working with the git repository which the bot posts about

The Plan

How the bot should work was rather clear from the outset. First of all, it shouldn't be a always-running-process. I wanted it to be as simple as possible, thus, triggering it via a systemd-timer should suffice. Next, it should only fetch the latest commits, so it should be able to work on a working clone of a repository. This way, we don't need another clone of a potentially huge repository on our disk. The path of the repository should of course not be hardcoded, as shouldn't the “upstream” remote name or the “master” branch name (because you might want to track a “release-xyz” branch or because “master” was renamed to something else).

Also, it should be configurable how many hours of commits should be checked. Maybe the user wants to run this bot once a day, maybe once a week. Both is possible, of course. But if the user runs it once a day, they want to check only the commits of the last 24 hours. If they run it once a week, the last 168 hours would be more appropriate.

The message that gets posted should also not be hardcoded, but a template where the variables the bot counted are available.

All the above goes into the configuration file the bot ready (and which can be set via the --config option on the bots CLI).

The configuration struct for the setup described above is rather trivial, as is the CLI setup.

The setup

The first things the bot has to do is reading the commandline and the configuration after initializing the logger, which is a no-brainer, too:

fn main() -> Result<()> {
    env_logger::init();
    log::debug!("Logger initialized");

    let opts = Opts::from_args_safe()?;
    let config: Conf = {
        let mut config = ::config::Config::default();

        config
            .merge(::config::File::from(opts.config().to_path_buf()))?
            .merge(::config::Environment::with_prefix("COMBOT"))?;
        config.try_into()?
    };
    let mastodon_data: elefren::Data = toml::de::from_str(&std::fs::read_to_string(config.mastodon_data())?)?;

The mastodon data is read from a configuration file that is different from the main configuration file, because it may contain sensitive data and if a user wants to put their configuration of the bot into a (public?) git repository, they might not want to include this data. That's why I opted for another file here, its format is described in the configuration example file (next to the setting where the file actually is).

Next, the mastodon client has to be setup and the repository has to be opened:

    let client = elefren::Mastodon::from(mastodon_data);
    let status_language = elefren::Language::from_639_1(config.status_language())
        .ok_or_else(|| anyhow!("Could not parse status language code: {}", config.status_language()))?;
    log::debug!("config parsed");

    let repo = git2::Repository::open(config.repository_path())?;
    log::debug!("Repo opened successfully");

which is rather trivial, too.

The Calculations

Then, we fetch the appropriate remote branch and count the commits:

    let _ = fetch_main_remote(&repo, &config)?;
    log::debug!("Main branch fetched successfully");

    let (commits, merges, nonmerges) = count_commits_on_main_branch(&repo, &config)?;
    log::debug!("Counted commits successfully");

    log::info!("Commits    = {}", commits);
    log::info!("Merges     = {}", merges);
    log::info!("Non-Merges = {}", nonmerges);

The functions called in this snippet will be described later on. Just consider them working for now, and let's move on to the status posting part of the bot now.

First of all, we use the variables to compute the status message using the template from the configuration file.

    {
        let status_text = {
            let mut hb = handlebars::Handlebars::new();
            hb.register_template_string("status", config.status_template())?;
            let mut data = std::collections::BTreeMap::new();
            data.insert("commits", commits);
            data.insert("merges", merges);
            data.insert("nonmerges", nonmerges);
            hb.render("status", &data)?
        };

Handlebars is a perfect fit for that job, as it is rather trivial to use, albeit a very powerful templating language is used. The user could, for example, even add some conditions to their template, like if there are no commits at all, the status message could just say “I'm a lonely bot, because nobody commits to master these days...” or something like that.

Next, we build the status object we pass to mastodon, and post it.

        let status = elefren::StatusBuilder::new()
            .status(status_text)
            .language(status_language)
            .build()
            .expect("Failed to build status");

        let status = client.new_status(status)
            .expect("Failed to post status");
        if let Some(url) = status.url.as_ref() {
            log::info!("Status posted: {}", url);
        } else {
            log::info!("Status posted, no url");
        }
        log::debug!("New status = {:?}", status);
    }

    Ok(())
} // main()

Some logging is added as well, of course.

And that's the whole main function!

Fetching the repository.

But we are not done yet. First of all, we need the function that fetches the remote repository.

Because of the infamous git2 library, this part is rather trivial to implement as well:

fn fetch_main_remote(repo: &git2::Repository, config: &Conf) -> Result<()> {
    log::debug!("Fetch: {} / {}", config.origin_remote_name(), config.master_branch_name());
    repo.find_remote(config.origin_remote_name())?
        .fetch(&[config.master_branch_name()], None, None)
        .map_err(Error::from)
}

Here we have a function that takes a reference to the repository as well as a reference to our Conf object. We then, after some logging, find the appropriate remote in our repository and simply call fetch for it. In case of Err(_), we map that to our anyhow::Error type and return it, because the callee should handle that.

Counting the commits

Counting the commits is the last part we need to implement.

fn count_commits_on_main_branch(repo: &git2::Repository, config: &Conf) -> Result<(usize, usize, usize)> {

The function, like the fetch_main_remote function, takes a reference to the repository as well as a reference to the Conf object of our program. It returns, in case of success, a tuple with three elements. I did not add strong typing here, because the codebase is rather small (less than 160 lines overall), so there's not need to be very explicit about the types here.

Just keep in mind that the first of the three values is the number of all commits, the second is the number of merges and the last is the number of non-merges.

That also means:

tuple.0 = tuple.1 + tuple.2

Next, let's have a variable that holds the branch name with the remote, like we're used from git itself (this is later required for git2). Also, we need to calculate the timestamp that is the lowest timestamp we consider. Because our configuration file specifies this in hours rather than seconds, we simply * 60 * 60 here.

    let branchname = format!("{}/{}", config.origin_remote_name(), config.master_branch_name());
    let minimum_time_epoch = chrono::offset::Local::now().timestamp() - (config.hours_to_check() * 60 * 60);

    log::debug!("Branch to count     : {}", branchname);
    log::debug!("Earliest commit time: {:?}", minimum_time_epoch);

Next, we need to instruct git2 to create a Revwalk object for us:

    let revwalk_start = repo
        .find_branch(&branchname, git2::BranchType::Remote)?
        .get()
        .peel_to_commit()?
        .id();

    log::debug!("Starting at: {}", revwalk_start);

    let mut rw = repo.revwalk()?;
    rw.simplify_first_parent()?;
    rw.push(revwalk_start)?;

That can be used to iterate over the history of a branch, starting at a certain commit. But before we can do that, we need to actually find that commit, which is the first part of the above snippet. Then, we create a Revwalk object, configure it to consider only the first parent (because that's what we care about) and push the rev to start walking from it.

The last bit of the function implements the actual counting.

    let mut commits = 0;
    let mut merges = 0;
    let mut nonmerges = 0;

    for rev in rw {
        let rev = rev?;
        let commit = repo.find_commit(rev)?;
        log::trace!("Found commit: {:?}", commit);

        if commit.time().seconds() < minimum_time_epoch {
            log::trace!("Commit too old, stopping iteration");
            break;
        }
        commits += 1;

        let is_merge = commit.parent_ids().count() > 1;
        log::trace!("Merge: {:?}", is_merge);

        if is_merge {
            merges += 1;
        } else {
            nonmerges += 1;
        }
    }

    log::trace!("Ready iterating");
    Ok((commits, merges, nonmerges))
}

This is done the simple way, without making use of the excelent iterator API. First, we create our variables for counting and then, we use the Revwalk object and iterate over it. For each rev, we unwrap it using the ? operator and then ask the repo to give us the corresponding commit. We then check whether the time of the commit is before our minimum time and if it is, we abort the iteration. If it is not, we continnue and count the commit. We then check whether the commit has more than one parent, because that is what makes a commit a merge-commit, and increase the appropriate variable.

Last but not least, we return our findings to the caller.

Conclusion

And this is it! It was a rather nice journey to implement this bot. There isn't too much that can fail here, some calculations might wrap and result in false calculations. Possibly a clippy run would find some things that could be improved, of course (feel free to submit patches).

If you want to run this bot on your own instance and for your own repositories, make sure to check the README file first. Also, feel free to ask questions about this bot and of course, you're welcome to send patches (make sure to --signoff your commits).

And now, enjoy the first post of the bot.

tags: #mastodon #bot #rust

Today, I challenged myself to write a prometheus exporter for MPD in Rust.

Shut up and show me the code!

Here you go and here you go for submitting patches.

The challenge

I recently started monitoring my server with prometheus and grafana. I am no-way a professional user of these pieces of software, but I slowly got everything up and running. I learned about timeseries databases at university, so the basic concept of prometheus was not new to me. Grafana was, though. I then started learning about prometheus exporters and how they are working and managed to setup node exporters for all my devices and imported their metrics into a nice grafana dashboard I downloaded from the official website.

I figured, that writing an exporter would make me understand the whole thing even better. So what would be better than exporting music data to my prometheus and plotting it with grafana? Especially because my nickname online is “musicmatze”, right?

So I started writing a prometheus exporter for MPD. And because my language of choice is Rust, I wrote it in Rust. Rust has good libraries available for everything I needed to do to export basic MPD metrics to prometheus and even a prometheus exporter library exists!

The libraries I decided to use

Note that this article was written using prometheus-mpd-exporter v0.1.0 of the prometheus-mpd-exporter code. The current codebase might differ, but this was the first working implementation.

So, the scope of my idea was set. Of course, I needed a library to talk to my music player daemon. And because async libraries would be better, since I would essentially write a kind of a web-server, it should be async. Thankfully, async_mpd exists.

Next, I needed a prometheus helper library. The examples in this library work with hyper. I was not able to implement my idea with hyper though (because of some weird borrowing error), but thankfully, actix-web worked just fine.

Besides that I used a bunch of convenience libraries:

  • anyhow and thiserror for error handling
  • env_logger and log for logging
  • structopt for CLI parsing
  • getset, parse-display and itertools to be able to write less code

The first implementation

The first implementation took me about four hours to write, because I had to understand the actix-web infrastructure first (and because I tried it with hyper in the first place, which did not work for about three of that four hours).

The boilerplate of the program includes

  • Defining an ApplicationError type for easy passing-around of errors that happen during the runtime of the program
  • Defining an Opt as a commandline interface definition using structopt
#[actix_web::main]
async fn main() -> Result<(), ApplicationError> {
    let _ = env_logger::init();
    log::info!("Starting...");
    let opt = Opt::from_args();

    let prometheus_bind_addr = format!("{}:{}", opt.bind_addr, opt.bind_port);
    let mpd_connect_string = format!("{}:{}", opt.mpd_server_addr, opt.mpd_server_port);

The main() function then sets up the logging and parses the commandline arguments. Thanks to env_logger and structopt, that's easy. The main() function also acts as the actix_web::main function and is async because of that. It also returns a Result<(), ApplicationError>, so I can easily fail during the setup phase of the program.

Next, I needed to setup the connection to MPD and wrap that in a Mutex, so it can be shared between request handlers.

    log::debug!("Connecting to MPD = {}", mpd_connect_string);
    let mpd = async_mpd::MpdClient::new(&*mpd_connect_string)
        .await
        .map(Mutex::new)?;

    let mpd = web::Data::new(mpd);

And then setup the HttpServer instance for actix-web, and run it.

    HttpServer::new(move || {
        App::new()
            .app_data(mpd.clone()) // add shared state
            .wrap(middleware::Logger::default())
            .route("/", web::get().to(index))
            .route("/metrics", web::get().to(metrics))
    })
    .bind(prometheus_bind_addr)?
    .run()
    .await
    .map_err(ApplicationError::from)
} // end of main()

Now comes the fun part, tho. First of all, I have setup the connection to MPD. In the above snippet, I add routes to the HttpServer for a basic index endpoint as well as for the /metrics endpoint prometheus fetches the metrics from.

Lets have a look at the index handler first, to get a basic understanding of how it works:

async fn index(_: web::Data<Mutex<MpdClient>>, _: HttpRequest) -> impl Responder {
    HttpResponse::build(StatusCode::OK)
        .content_type("text/text; charset=utf-8")
        .body(String::from("Running"))
}

This function gets called every time someone accesses the service without specifying an endpoint, for example curl localhost:9123 would result in this function being called.

Here, I can get the web::Data<Mutex<MpdClient>> object instance that actix-web handles for us as well as a HttpRequest object to get information about the request itself. Because I don't need this data here, the variables are not bound (_). I added them to be able to extend this function later on easily.

I return a simple 200 (that's the StatusCode::OK here) with a simple Running body. curling would result in a simple response:

$ curl 127.0.0.1:9123
Running

Now, lets have a look at the /metrics endpoint. First of all, the signature of the function is the same:

async fn metrics(mpd_data: web::Data<Mutex<MpdClient>>, _: HttpRequest) -> impl Responder {
    match metrics_handler(mpd_data).await {
        Ok(text) => {
            HttpResponse::build(StatusCode::OK)
                .content_type("text/text; charset=utf-8")
                .body(text)
        }

        Err(e) => {
            HttpResponse::build(StatusCode::INTERNAL_SERVER_ERROR)
                .content_type("text/text; charset=utf-8")
                .body(format!("{}", e))
        }
    }
}

but here, we bind the mpd client object to mpd_data, because we want to actually use that object. We then call a function metrics_handler() with that object, wait for the result (because that function itself is async, too), and match the result. If the result is Ok(_), we get the result text and return a 200 with the text as the body. If the result is an error, which means that fetching the data from MPD somehow resulted in an error, we return an internal server error (500) and the error message as body of the response.

Now, to the metrics_handler() function, which is where the real work happens.

async fn metrics_handler(mpd_data: web::Data<Mutex<MpdClient>>) -> Result<String, ApplicationError> {
    let mut mpd = mpd_data.lock().unwrap();
    let stats = mpd.stats().await?;

    let instance = String::new(); // TODO

First of all, we extract the actual MpdClient object from the web::Data<Mutex<_>> wrapper. Them, we ask MPD to get some stats() and wait for the result.

After that, we create a variable we don't fill yet, which we later push in the release without solving the “TODO” marker and when we blog about what we did, we feel ashamed about it.

Next, we create Metric objects for each metric we record from MPD and render all of them into one big String object.

    let res = vec![
        Metric::new("mpd_uptime"      , stats.uptime      , "The uptime of mpd", &instance).into_metric()?,
        Metric::new("mpd_playtime"    , stats.playtime    , "The playtime of the current playlist", &instance).into_metric()?,
        Metric::new("mpd_artists"     , stats.artists     , "The number of artists", &instance).into_metric()?,
        Metric::new("mpd_albums"      , stats.albums      , "The number of albums", &instance).into_metric()?,
        Metric::new("mpd_songs"       , stats.songs       , "The number of songs", &instance).into_metric()?,
        Metric::new("mpd_db_playtime" , stats.db_playtime , "The database playtime", &instance).into_metric()?,
        Metric::new("mpd_db_update"   , stats.db_update   , "The updates of the database", &instance).into_metric()?,
    ]
    .into_iter()
    .map(|m| {
        m.render()
    })
    .join("\n");

    log::debug!("res = {}", res);
    Ok(res)
}

Lastly, we return that String object from our handler implementation.

The Metric object implementation my own, we'll focus on that now. It will help a bit with the interface of the prometheus_exporter_base API interface.

But first, I need to explain the Metric type:

pub struct Metric<'a, T: IntoNumMetric> {
    name: &'static str,
    value: T,
    description: &'static str,
    instance: &'a str,
}

The Metric type is a type that holds a name for a metric, its value and some description (and the aforementioned irrelevant instance). But because the metrics we collect can be of different types (for example a 8-bit unsigned integer u8 or a 32-bit unsigned integer u32), I made that type abstract over it. The type of the metric value must implement a IntoNumMetric trait, though. That trait is a simple helper trait:

use num_traits::Num;
pub trait IntoNumMetric {
    type Output: Num + Display + Debug;

    fn into_num_metric(self) -> Self::Output;
}

And I implemented it for std::time::Duration, u8, u32 and i32 – the implementation itself is trivial and I won't show it here.

Now, I was able to implement the Metric::into_metric() function shown above:

impl<'a, T: IntoNumMetric + Debug> Metric<'a, T> {
    // Metric::new() implementation, hidden here

    pub fn into_metric<'b>(self) -> Result<PrometheusMetric<'b>> {
        let instance = PrometheusInstance::new()
            .with_label("instance", self.instance)
            .with_value(self.value.into_num_metric())
            .with_current_timestamp()
            .map_err(Error::from)?;

        let mut m = PrometheusMetric::new(self.name, MetricType::Counter, self.description);
        m.render_and_append_instance(&instance);
        Ok(m)
    }
}

This function is used for converting a Metric object into the appropriate PrometheusMetric object from prometheus_exporter_base.

The implementation is, of course, also generic over the type the Metric object holds. A PrometheusInstance is created, a label “instance” is added (empty, you know why... :–( ). Then, the value is added to that instance using the conversion from the IntoNumMetric trait. The current timestamp is added as well, or an error is returned if that fails.

Last but not least, a new PrometheusMetric object is created with the appropriate name and description, and the instance is rendered to it.

And that's it!

Deploying

The code is there now. But of course, I still needed to deploy this to my hosts and make it available in my prometheus and grafana instances.

Because I use NixOS, I wrote a nix package definition and a nix service defintion for it, making the endpoint available to my prometheus instance via my wireguard network.

After that, I was able to add queries to my grafana instance, for example:

mpd_db_playtime / 60 / 60 / 24

to display the DB playtime of an instance of my MPD in days.

I'm not yet very proficient in grafana and the query language, and also the service implementation is rather minimal, so there cannot be that much metrics yet.

Either way, it works!

A basic dashboard for MPD stats

Next steps and closing words

The next steps are quite simple. First of all, I want to make more stats available to prometheus. Right now, only the basic statistics of the database are exported.

The async_mpd crate makes a lot of other status information available.

Also, I want to get better with grafana queries and make some nice-looking graphs for my dashboard.

Either way, that challenge took me longer than I anticipated in the first place (“I can hack this in 15 minutes” – famous last words)! But it was fun nonetheless!

The outcome of this little journey is on crates.io and I will also submit a PR to nixpkgs to make it available there, too.

If you want to contribute patches to the sourcecode, which I encourage you to do, feel free to send me patches!

tags: #prometheus #grafana #rust #mpd #music

Holy crap, I haven't written on my blog for a long time. And I almost missed that the Rust community asked for blog posts about Rust in 2021 - but I am in time I guess, so here it goes.

Most Rustaceans won't agree with this blog post, I guess. But I also think that's fine, because that's the whole point of the Blog-Post-For-The-Roadmap thing, right? Asking people for different opinions and starting a constructive discussion about the topic. I also must say that I haven't read a single one of the other Rust-2021 Blog posts just yet.

I also think this will be rather short, but I hope I express my feelings in the best way possible for you all to understand.

Don't Change!

I got into Rust at about Rust 1.5.0. After the first half of 2020, I felt like January was years ago, so I feel like Rust 1.5.0 was in another lifetime. So much happened this year, and still, so little was accomplished by me and my friends. The world turned upside down, essentially.

Rust changed a lot between 1.5.0 and the current compiler I have installed on my system:

$ rustc --version
rustc 1.46.0 (04488afe3 2020-08-24)

The RELEASES.md file is a whooping 9167 lines long. We got cargo workspaces, we got awesome things like the ? operator (which I definitively was not a friend of in the beginning), we got associated constants, incremental compilation, impl Trait, we got const functions and most importantly we got async/await.

Fairly, that's where I started to struggle to keep up. I definitively see the value in async-await and what it actually enables us to do with Rust, and how to do it. But I just couldn't keep up with the change anymore. It was too much. I couldn't cope learning all these new things just in time they arrived. I, to this day, struggle to write a simple Program with async/await if there's too much iterators involved. I don't know where my actual problems are, because I cannot see through the whole concept enough to understand what I am doing wrong.

The last five years were full of change. Good change, of course. But this last (almost)year just drowned me. Too much to handle.

My hope is, that Rust does not change anymore when it comes to features. I see that there is a lot of demand for const generics, especially by the embedded community. I understand why. I hope it doesn't have any impact on me as a commandline-program-writing Rustacean.

But...

But. There's always a “but”, isn't there?

I still have high hopes for some things concerning Rust. But they do not at all have to do with the language Rust, but the environment around it. As stated before, I'm a commandline-program writing person. I do not write web services (yet?), I do not write embedded stuff, I do not write high-performance/performance-critical stuff.

Essentially: I write programs in Rust that others would write in Python, Ruby or Node. I write them in Rust because I am a lazy programmer, because I do not care enough. I write Rust, because the compiler YELLS at me to get it right. If I would do the same thing in Ruby, my go-to-language for everything below 100LOC, I would get myself into a wheelchair because I would use every footgun available.

Deep inside, I'm a bad programmer and rustc forces me to be a good one.

That was a bit of a rant, I hope you're still with me. The paragraph above was for you to understand where I come from. I don't care if my program runs in 1 second or 10, because the domain I write for does not care most of the time. But what is important to me, is that I actually can write my programs. Often, I cannot. And that's simply because of one thing:

Libraries are missing.

Those who know me knew that this was coming. Libraries for domains that I care about are missing. That is calendar (icalendar) reading/writing, vcard reading/writing, email reading/writing (the format, not the networking stuff), ... There are already libraries out there for these things, although they are far from being complete, usable or even correct. Writing a simple TUI MUA for notmuch is pain right now, because parsing email is really hard in Rust, and there are no high-level libraries available. The “mail” crate ecosystem is closest, but they do not yet have a parser.

There are, I am sure, more things in this part of the ecosystem (that is libraries for basic formats) where Rust could shine, but does not yet.

My request for Rust in 2021 is: Make things shiny. Make them available, make them work, make them correct, make them nice to use (E.G. parsing mails into tokens and handing them to me is okay, but having a high-level interface is much nicer).


To sum it up in one sentence:

Don't change rust itself, but improve the library ecosystem.

That's my hope for Rust 2021. Thank you for having me in this awesome community and thank you for reading.

tags: #rust #programming

Finally, I managed to implement a proof of concept of serde-select. But lets start at the beginning.

The Problem

The problem I tried to solve with this crate is rather simple: You need to be able to get values from a serde-compatible document (e.g. toml, json, yaml, ...) but you don't know the full schema of the document at compiletime of your crate.

The origin of the idea of serde-select was when I first started working on my imag project, where a lot of seperated crates coexist in one ecosystem, but all of them should be configured in one big configuration file. Of course I did not want to have one central crate just for defining the schema, especially since a user might not want to use all functionality from the ecosystem, thus not having a “full” configuration file, but only the parts they needed.

So I started writing “toml-query”, a crate which lets the programmer query a toml::Value with a “path”. For example:

[calendar]
list_format = "{{lpad 5 i}} | {{abbrev 5 uid}} | {{summary}} | {{location}}"
show_format = """
{{i}} - {{uid}}
"""

[ref]
[ref.basepathes]
music = "/home/user/music"
contacts = "/home/user/contacts"
calendars = "/home/user/calendars"

The document looks like this, but in the program code we only need calendar and its sub-values. So we can do

let r = document.read("calendar.list_format");

in the code and get a Result<Option<&'document Value>> value back.

toml-query evolved over the time, now featuring more flexibility by implementing “Partials”, how I call them. These are structs that are Serialize + Deserialize and have a path attached to them, so deserializing the partial document is possible right away:

let r: Result<Option<CalendarConfig>, _> = document.read_partial::<CalendarConfig>();

where CalendarConfig: Serialize + Deserialize + Debug + toml_query::Partial (see here).

The evolution

toml-query works perfectly fine and I use it in my other projects a lot. It is fast and easy to use. Its error reporting is nice.

But an idea formed in the back of my head and I did not stop to think about it.

Can toml-query by generalized to work with all formats serde can handle?

So I started to experiement with a more general implementation: serde-select was born.

And today I managed to get the first bits working.

Meet serde-select

serde-select implements a “read” functionality for both JSON and TOML, depending on what features you enable. The inner implementation of the resolve-algorithm is agnostic of the actual format.

For a quick overview how to use the crate right now, have a look at the tests for toml for example.

I strongly advice against using this crate, though. It is only an experiement for now and shouldn't be used in production code. Nevertheless, I published the first preview on crates.io.

tags: #rust #programming

The call for blogs was just issues a few days ago – and here I am writing about my biggest pains this year... because that's what the call for blogs basically is for me... I write down my pains with Rust and hope things get better slowly next year.

Don't misunderstand what I want to say here though: Rust is awesome, has an awesome community, awesome tooling, awesome everything... well not completely (because otherwise I wouldn't have to write this article, right?), but almost.

Pain #0: Libraries

Pain Number Zero (because that's where computer programmers start to count) is the library ecosystem. “What?” you say? “Rust is known for a really good library ecosystem although the language is not even five years old (counting from 1.0.0)“, you might say! And of course you're right... but not in my domain, unfortunately.

Rust has excellent libraries for developing web services backend as well as backend, gaming engines and games and of course microcontroller stuff. But these are not my domains. My domain is commandline user stuff. My domain might become TUI applications or even GUI applications in the future. My domain is data formats, especially icalendar and vcard, because I write journal applications, calendar applications, contact management stuff, todo applications and diary tooling and even Email processing/handling stuff and possibly even a CLI/TUI mail reader – of course I'm talking about imag here, the text-based, commandline personal information management suite I'm developing over four years now.

The tooling in this domain is not nonexistent, no way! But, despite the efforts some people in this awesome community started, the number and especially the quality of those libraries is nowhere as satisfying as the support for other domains. No offense to the libraries authors of course! It is not their fault at all. It is just that only a few people have started initiatives in this direction yet. I try to contribute! I am actually working on a libical high-level frontend which I started to extract from khaleesi, a work by two wonderful people which includes a wrapper around libical and libical-sys that I started to extract into a library crate.

But there could be so much more and better support for these things! I can only do so much. So I call out: Help developing libraries for these standards! Especially help developing high-level Rust libraries for these things, because handling mime as a way to work with emails is just the beginning. Parsing mail into something that can be worked with on a high level in Rust would be a wonderful goal for 2020. And of course all the other domains!

I remember from my days with Ruby that code could be written at a high level when working with mail and other such formats. Lets have that in Rust!

Lets have world-class support for handling data formats at a high level, so we can write “Speaking code” like it would be plaintext.

Pain #1 – CLI

My next pain are frontends. What I mean by that is CLI, TUI and GUI frontends, not WUI (web user interface) frontends. But I'll break this down into several sections here, so let's talk about CLI first...

One thing here is of course the wonderful clap crate. Lets make clap v3.0 happen next year! It would be a huge step forward!

But this is only one minor pain point, because clap is already a wonderful thing. Lets also make the interactive commandline user interface story better!

I remember that the people from the Node community have commandline applications that you can use interactively that just amaze me because they are so comfy to use (and I'm not even talking about TUI applications here, just interactive CLI apps)! I think we can have this in Rust!

Lets have the best libraries to implement interactive commandline applications!

Pain #2 – TUI

TUI is the next thing i want to point out. Short disclaimer though: I never wrote a TUI app, but I certainly plan to do so. Maybe not in 2020 but after that, imag should get a TUI interface at some point. And for that, of course, I would love to have a headache after thinking about which library or framework to chose.

Right now, there's cursive – and holy swearword this thing looks amazing! But there could be so much more, still! Of course there are already a few extension crates out there:

(btw: @deinstapel you're a hero – they implemented half of the crates above!)

But I bet there could be more... I could think of an embedded terminal for cursive, I can think of an editor-view for embedding vim or another TUI editor into a cursive application, I could think of an editor-like view embedding the Xi editor...

Lets make Rust the go-to choice for writing TUI applications!

Pain #3 – GUI

And of course, the GUI domain. I could write up a long text here, or just point you to other Rust-2020 articles that expand on the topic... but I don't. Why? Because I never implemented a GUI application, I don't see myself implementing one in the near (or even far) future (at least not for imag) and so I don't take the liberty to reiterate what others said more eloquently: The Rust ecosystem for writing GUI applications is not good.

Lets improve our GUI-writing experience!

Summary

All in all, I hope that 2020 will be the year of the Rust Language as application language. We have an awesome tooling and frameworks available for web stuff, the game-domain is expanding constantly and low-level programming is possible and done out there all the time.

Writing applications in Rust is not yet as awesome as it could be, though. So my hopes, dreams and wishes are...

Lets make Rusts high-level application writing experience the best out there!

tags: #rust

This is a reply to the article published by Drew DeVault called “Rust is not a good C replacement”.


First of all, let me say that Drew is one of the people out there on the internet whose opinions I highly value. Indeed he is one of the people I try to read and listen to regardless of topic, because I think he is one of the people that deserve unconditional attention.

Needless to say that I've also read his latest piece “Rust is not a good C replacement”. I have to admit I was shocked at first, but after a bit of cooling down (and doing the dishes), I can see where this comes from.

And I have to disagree.

But let me start with my background – because that might be important for you, dear reader, to classify this article.

My background is mostly hobbyist programming. I did a few years of C, probably a few 100kloc, not more. I also do rust since about Rust 1.5.0 (2015-12). I started a job where I expect to write C and C++ professionally about 1 month ago.

So, I do not have a background like Drew with probably millions of lines of C, but I guess that I have a bit more experience with Rust – I wouldn't say that I'm a Rust professional, but I would consider myself a “Advanced Rust Hobbyist”.

I'm also not as skilled in writing blog articles or even with the english language, so keep that in mind when reading this.


I am not a big fan of statement-by-statement replying to an article, but I guess for this type of article it is good enough.

First of all, Drews initial statement that Rust was designed by C++ programmers: Yes, I absolutely see that this is true. Nevertheless I have to say that these C++ programmers started developing Rust because C++ was too complex and too error-prone in what it did and how it did it. Rust is far away from the complexity C++ gives us in terms of language features! From the top of my head, we have

  • A full-blown OOP programming paradigm, including
    • Overloading
    • “friends”
    • (multi-)inheritance
    • abstract classes
    • partly and fully virtual functions
    • pointers and references
    • implicit conversions
    • Copy/Move constructors
    • Dynamic and static polymorphism
  • Manual memory management
  • Template Metaprogramming / Generic programming
  • operator overloading
  • Lambda expressions
  • Exceptions

in C++, whereas in Rust we only get

  • Dynamic and static polymorphism
  • operator overloading
  • Lambda expressions
  • Generic programming[^1]

([^2])

You might consider this list cheated as Rust is not an object oriented language like C++, but an imperative one like C. That is very true. Nevertheless it is one reason why the cognitive load a C++-Program requires one to handle is much higher than an equivalent (as in features of the program) Rust program!

Drew claims that the values of C and C++ programmers are incompatible and I would agree with that. But that does not (have to) mean that a C-programmer and a Rust programmer do not have the same values. It is true, though, that Rust can excel at a lot of topics that C++ covers, but it also empower programmers that do not feel comfortable writing good C code to write their software. And that in a safe and at the same time performant language while not being overly blown up.

Further Drew compares C, C++, Go and Rust by their complexity, measured with features introduced in the language over the years. I am really sorry to say this here, Drew, but we are used to much better from you! You say that this approach (bullet points/features listed on wikipedia vs. bullet points in articles and release notes) is not very scientific, yes. But not even to mention the years these languages were released! For the record:

I am not saying that this disproves your statement – it even supports it! But I do say that comparing based on features per year/release/whatever must include a statement about how old these languages are, even if it is just for showing the reader about what timeframe we are talking.

So, Rust is a relatively young language and yes, it has added a lot of features and therefore can be compared to C++ much better than it can be to C. But saying that a 10-year-old C programm might even compile today and everything might be okay but not so much with Rust is just ignorant of the fact that Rust is not even that old. Having a Rust program that is one year old still compiles fine today (assuming it didn't use a compiler bug) and does not “look old” at all! Rust has a big infrastructure for doing regression tests and for being compatible with older programs.

As you say, out of the way with the philosophical stuff and lets get down to the facts.

C is more portable. But as mentioned before, C is almost six times as old as Rust. We'll get there!

C has a spec. Yes, and I completely hear you on this one. Rust does not (yet?) have a spec and it really is a pain-point. I want one, too! Maybe we'll get there at some point. By the way: Does Go have a spec? It seems like it, but that rather looks like a language definition and I doubt that this is what Drew meant when talking about “a spec”, is it?

C has many implementations. Yes and how much trouble has it been because different compilers do different things on undefined behaviour? Too many. This is where Rust tries to solve a problem: Get a language where undefined behaviour is not allowed or at least as minimal as possible, then we can have a spec for that language and then we can have different implementations. Time will tell whether we can get there.

C has a consistent & stable ABI. Point taken. I do not argue about that.

Cargo is mandatory. Yes, another point taken. I again do not argue.

Concurrency is generally a bad thing. This statement gives me the impression that you did not yet try Rust, actually. Like in a big (and possibly multithreaded/concurrent/parallel/whateveryoucallit) environment. You say that most software does not have to be parallel and I fully agree on that – but if you need to be parallel, I'd rather chose Rust over Go, C or C++. Having the safety guarantees Rust gives me allows normal people (and not Rockstar-programmers) to write software that can be massively parallel without having to fear about deadlocks and other ugly things you get with other languages.

It is still true that bad design decisions are possible and might result in bad software – but that is true for every language, isn't it? And I'd rather like to have a bad program that gets the job done because it can be statically verified that it does than a program that crashes because I ran into a bug that was introduced by bad design decisions.

The next paragraph Drew writes makes me really, really sad. Fullquote:

Safety. Yes, Rust is more safe. I don’t really care. In light of all of these problems, I’ll take my segfaults and buffer overflows. I especially refuse to “rewrite it in Rust” – because no matter what, rewriting an entire program from scratch is always going to introduce more bugs than maintaining the C program ever would. I don’t care what language you rewrite it in.

This gives me the impression that Drew was hit with “Just rewrite it” too many times. And I completely agree with you, Drew, that you should indeed not rewrite it in Rust just for the sake. Nobody should ever rewrite anything in any other language than what “it” currently is written in. I hate these people that actually say things like that (if it isn't for trolling, but I have the uneasy feeling that Drew was hit with real “Just rewrite it”ers and not just trolling).

I do not say that the points Drew shows are false.

What I do say is that the initial assumption that Rust is there to replace C or C++ is, in my opinion, false. It is certainly meant to get things right that C++ got wrong – and it is certainly there to replace the C++-Monster that we call Gecko, because Mozilla is exactly trying to do that! But it is not there to replace all C or C++ code ever written because of some stupid “Hey we can do X better than your language” bullshit!

Also, the statement that Rust might end up as Kitchen-Sink like C++ and die with feature-bloat is one that concerns me because I do not want Rust to end up like C++. It certainly is not as complex as C++ and we (as in “the Rust community”) have a lot of work to do to not end up with feature-creep – but we are also certainly not there yet. But I definitively see where this statement is coming from.

The title of this article is “Rust is one of the best C replacements we currently have” – and I stand by this. But I also think that it is false to say that anyone has to replace C or that Rust is necessarily there to do so.

There are domains where you might want to rewrite C code, if you have the time and resources. But I'd rather advice against it[^3]. Improving existing code is always easier than a rewrite of a program and rewriting software does not improve the value of the software or even make customers more happy. Rewriting software is IMHO only legit in two cases:

  • It makes you happy because you're doing it for fun
  • It makes your boss happy because he ordered you to do so (for whatever reasons, may it be speed, resource usage, customer request or whatever)

But just for the sake of it? Nah.

I see where Drews article comes from and I see why he thinks like he does. I greatly value his opinion and thoughts, and that's why I took the time to write this article.

I see that we (as in “the Rust community”) have a lot to do to make more people happy. Not as in making them Rust programmers, because that's not our goal, but as in showing them that we do not want everything to be written in Rust and that it is just trolls that request a “rewrite in Rust”.

We do value friendlyness and kindness – let me state explicitely that this does also include other programming-language-communities (and all other communities as well)!

Trolling does not help with that.

[^1]: Yes we have generic programming in Rust. I'm not a professional regarding C++, so I cannot say whether they are comparable in this regard. [^2]: Some might say that we have manual memory management in Rust as well. That might be true by definition, but not the way I meant it: In C++ we can allocate something on the heap and then forget it. We have to try really hard to do that in Rust, though! [^3]: In fact I might get into the situation where I have to rewrite an application in my job, but I'd rather rewrite it in the same language than switching languages just for the sake of it!

tags: #open-source #programming #rust #c #c++ #cpp

This post was written during my trip through Iceland and published much later than it was written.

This is a really important topic in programming and I really hope to get this article right. Not only for technical correctness, but also for ease to understand, as explaining types is not that simple if one has never heard of them.

Let's give it a try...

What are types

Well, that's a question which is, in my opinion, not easy to answer. In fact, I thought several days about this question before writing this down, in hope it will become a sufficient answer. Hence, you might find other answers which are easier to understand and maybe more correct as mine, but I'll give it a try nonetheless.

From what I think

Types are a combination of abilities and properties that are combined to express and limit a certain scope of a thing.

For example, A type Car may have four wheels, two doors and a horn (its properties) and can drive slow, drive fast and park (its abilities). That is certainly not a real representation of a car (also because only a car is a real representation of a car) but because of the domain this is used in, it is sufficient in the scenario at hand. The type Car cannot be multiplied, but another type Number may have this ability. Thus, the scope and abilities are also limited in a certain way.

I hope this description is a good one for you to understand.

Now that we know what types are, we should also learn some other terms around the subject of types. The first thing I want to talk about here is “strong typing” and “weak typing”. The reason for this is: These things do not exist. Yes, you've read this correctly: There is no such thing as “strong typing”. There is only stronger and weaker typing. The Java programming language is not strongly typed, neither is it weakly typed (but it is, of course badly typed... forgive me that joke, pity java programmer).

But what is a stronger typing? That is rather simple to explain, actually. We discussed that types are limitations of things to be able to only do some specific operations and such. These things are enforced by the compiler or interpreter of the programming language, of course. And stronger typing only says that the compiler has more information (implicitly via the definition of the programming language) to enforce these rules, the rules of “the function A is defined for type T, so you cannot call it on U”. Of course there is more to that because of generic typing and so on, but that's basically it.

The next term is “type inference”. Type inference is nothing a programmer experiences explicitely, because it happens implicitly. Type inference is a feature of the compiler and interpreter of the language to guess the type of a variable without the programmer stating the actual type. It's nothing more to that actually.

I mentioned the term “generic types” in one of the former paragraphs already, so we should have a look there, too. Generic types, or shorter Generics, are types which are partial, in a way. So for example, one can define a Bag of things, whatever things is. This is often (at least in typed languages – languages where types actually matter for the compiler or interpreter) specified in the code via “type parameters” (though this term differs from language to language).

Why more types are better then few

The more types you introduce in your programs (internally or even for the public API), the more safety you get (speaking in the context of a stronger typed programming language, but also if you do a lot of runtime-type-checking in a weaker typed language). That does not mean that you should introduce a BlueCar, a BlackCar and a GreenCar as types in your program, but rather a type Color and a type Car whereas each Car has a Color – even if your domain is cars and not colors.

Maybe that example lacks a certain expressiveness, so consider this: Your Car has wheels. You can set the number of wheels when constructing the Car object. But instead of passing an integer here, which would yield an API where one can pass 17 as valid number for the number of wheels – or 1337 or possibly even -1. But if you introduce a type which represents the number of wheels, you get some safety into the construction of the Car object – safety checks in your code are not necessary anymore and thus your code will be shorter, better focused on what the actual problem is instead of fighting for valid values and of course, the compiler or interpreter can do the work for you.

Sounds nice, doesn't it? You can get this all with (almost) no cost attached, you just have to write down some more types. If your programming language contains feature like enumerations, you do not even have to make validity checks anymore, as the compiler can execute them.

Next

In the next post we will focus on the coding environment.

tags: #open-source #programming #software #tools #rust

This post was written during my trip through Iceland and published much latern than it was written.

While we heavily focused on the code-surrounding things in the last parts, we will return to focus on code-related things from here on.

This article discusses code verbosity and how it can improve your open source code and also your contributors experience a lot.

What is code verbosity

Code verbosity is mainly explicitness of code. For example, in Java you have to be more explicit when declaring a variable than in (recent) C++ or even Ruby:

String s = someFunctionCall(param); // Java

auto s = someFunctionCall(param); // C++

s = someFunctionCall param # Ruby

So code verbosity is how explicit you have to state certain things so that the compiler or interpreter understands your intention.

Because we do not always tell the compiler or interpreter what we want to do exactly and because we want to re-use functionality, we introduced abstractions. So abstractions are a way to make code less verbose, in some ways.

How to make code less verbose

Abstraction. It is as simple as this. You introduce abstraction to minimize repetition which leads to less verbose code. Of course, you cannot always make the code less verbose if the language does not allow it: in the above example we used the auto keyword for specifying the type in C++, which is nice, but not possible in Java. So in the borders of your languages abilities, you can make code less verbose.

If you do that right and the abstractions results in nice code, you know that you've done fine.

How much is too much

But there can also be too much abstraction which then yields unreadable code. Not unreadable as in clustered with stuff but just too abstract to grasp at first sight.

Abstraction can get too much. So make sure you introduce sensible abstractions, abstractions that can be combined nicely and of course one can step around the abstractions and use the core functionality, the not-abstracted things beneath.

As a sidenote: sometimes it makes sense to hide certain things completely or even introducing several layers of abstractions.

Next

This was a rather short one, I guess. The next article will be longer I hope, as it will be about typing.

tags: #open-source #programming #software #tools #rust