Epoch fail, Clojure edition

A while back, I wrote about doing epoch calculations in Elixir. Recently, I tried the same sort of thing in Clojure. Since Clojure sits on top of Java, I was not optimistic. Historically, Java has not had a good story to tell when it comes to date-time manipulation. But Java 1.8 added a new java.time package, so maybe it’s better now?

Clojure has clj-time, which wraps joda-time. Joda Time is the third-party library that most Java folks used before version 1.8 (and still do, I guess, if they’re stuck on 1.7 or 1.6 for some reason). Perhaps clj-time will move to java.time eventually, but for now at least they are still on joda-time. Maybe it would have been more sensible for me to just have used clj-time, but I wanted to use the java.time package.

I discovered there is a Clojure java-time (with a dash) that wraps java.time (with a period) and I started out doing that, but eventually abandoned it. When I got to the google-calendar calculation, I needed the .toInstant method, and java-time didn’t seem to supply it.

So, I changed to using java.time directly (if you look back in the git history of Epochs-clojure, you can see where I switched). Since Clojure’s Java interop is so seamless, it turned out pretty nice! Here is that google-calendar portion:

(defn google-calendar
  "Google Calendar time seems to count 32-day months from the day
  before the Unix epoch. @noppers worked out how to do this."
  [n]
  (let [seconds-per-day (* 24 60 60)
        utc (java.time.ZoneOffset/UTC)
        n-days (quot n seconds-per-day)
        n-seconds (rem n seconds-per-day)
        g-months (quot n-days 32)
        g-days (rem n-days 32)]
    (-> (java.time.LocalDateTime/ofEpochSecond (- seconds-per-day) 0 utc)
        (.plusDays g-days)
        (.plusMonths g-months)
        (.toInstant utc)
        (.plusSeconds n-seconds))))

Java time has Instants, which are the naive date-times that I want to do everything with. Similar to Elixir, java.time won’t let me add days or months to an Instant. So I create a LocalDateTime, add the days and months, then convert to an Instant and add the seconds.

Not bad. Not as nice as the Perl version, but better than the Elixir version. At least java.time gives me the .plusMonths method I need, even if it does make me choose a timezone to get it. And, remarkably, it’s just Java. I usually can’t bear to look at Java code, but Clojure makes it almost pretty.

I look forward to clj-time or java-time supplying more Clojure-like access to java.time in the future. But in the meantime, accessing it directly isn’t that bad!


(Update: 2017-08-06) I went to see if I could add .toInstant support for LocalDateTimes in java-time. I figured rather than just complain, I’d send a pull request. In so doing, I discovered it is already there! If I have a LocalDateTime in ldt, then this Java interop code

(.toInstant ldt java.time.ZoneOffset/UTC)

can be written as this java-time code

(instant ldt (zone-offset 0))

I think I’m going to leave Epochs-clojure as Java interop for now, but maybe I’ll switch back to java-time in the future if things start to get too Java-y.

Epoch fail, Clojure edition

Hello, interpolation!

You may have noticed in Command-line Clojure that we glued the string together in pieces

(println (str "Hello, " name "!"))

What are we, Neanderthals? There is a string format in Clojure, but that’s not much better

(println (format "Hello, %s!" name))

Doesn’t Clojure have interpolation? Apparently not, but Chas Emerick wrote a macro for it some years ago. According to that, we should be able to use

(println (<< "Hello, ~{name}!"))

instead. The “<<” is a macro that enables variable interpolation in the string with “~{}”. It does a whole lot more too. Nice!

So how do we use it? It’s not in the core of Clojure yet, but it has been moved into something called the core.incubator. It says there we just need to

Add a dependency on core.incubator to your Leinigen project.clj:

[org.clojure/core.incubator “0.1.4”]

So if we create a new project

$ lein new hello-world
$ cd hello-world

and then add that dependency

$ cat project.clj 
(defproject hello-world "0.1.0-SNAPSHOT"
  :description "Hello interpolation"
  :dependencies [[org.clojure/clojure "1.8.0"]
                 [org.clojure/core.incubator "0.1.4"]])

we should be able to use the macro.

$ cat src/hello_world/core.clj 
(ns hello-world.core)

(defn hello
  ([]
   (hello "World"))
  ([name]
   (<< "Hello, ~{name}!")))

But that doesn’t work. We still need to add the namespace. And not clojure.core.incubator as we might expect, but clojure.core.strint!

(ns hello-world.core
  (:use [clojure.core.strint]))

(defn hello
  ([]
   (hello "World"))
  ([name]
   (<< "Hello, ~{name}!")))

Ah, interpolation! Sweet, sweet, interpolation!

$ rlwrap lein repl
user=> (use 'hello-world.core)
user=> (hello)
"Hello, World!"
user=> (hello "Hank")
"Hello, Hank!"

I do hope they add the “<<” macro to the core soon!

Hello, interpolation!

Command line Clojure

I guess folks usually run Clojure in a REPL, but what if we wanted to write a command-line utility?

Clojure

That’s simple enough to do. If we had the following

#!/usr/bin/env clj

(println (str "Hello, World!"))

in an executable file hello.clj, then we could run it just like anything else.

$ ./hello.clj 
Hello, World!

But how do we interact with our environment? For example, most scripts need access to the command line arguments. In Clojure, these are provided in the *command-line-args* variable. So if we have the following

#!/usr/bin/env clj

(def args *command-line-args*)

(defn greet [name]
  (println (str "Hello, " name "!")))

(if (empty? args)
  (greet "World")
  (doseq [name args]
    (greet name)))

then we’ll greet each command line argument in turn.

$ ./hello.clj Hank Dean Brock
Hello, Hank!
Hello, Dean!
Hello, Brock!

If there are none, we’ll just greet “World” like before.

$ ./hello.clj 
Hello, World!

This is fine, but it’s kind of slow.

$ time ./hello.clj 
Hello, World!

real    0m2.278s
user    0m3.484s
sys     0m0.168s

I guess this is because we have to wait for the JVM to start up. I’m not sure we’d want a command line utility this slow. What can we do?

ClojureScript

How about ClojureScript? We normally think of that as running in the browser, but there are other ways to use it as well. Perhaps server-side JavaScript is faster to start up than Java.

ClojureScript with WebKit

Planck is a stand-alone ClojureScript REPL that is based on WebKit. We could write our script for that with just a small change

#!/usr/bin/env planck
;; -*- clojure -*-

(ns hello.core
  (:require [planck.core :refer [*command-line-args*]]))

(def args *command-line-args*)

(defn greet [name]
  (println (str "Hello, " name "!")))

(if (empty? args)
  (greet "World")
  (doseq [name args]
    (greet name)))

We have to add a line to get Planck to arrange for WebKit to get the *command-line-args* for us. Running this, we get

$ time ./hello-planck 
Hello, World!

real    0m3.318s
user    0m4.532s
sys     0m0.316s

Hrm. That didn’t help. What else?

ClojureScript with Node

Lumo is a stand-alone ClojureScript REPL based on Node.js. To use it, we change our script again

#!/usr/bin/env lumo
;; -*- clojure -*-

(def process (js/require "process"))

(def args (.slice (.-argv process) 3))

(defn greet [name]
  (println (str "Hello, " name "!")))

(if (empty? args)
  (greet "World")
  (doseq [name args]
    (greet name)))

Here we ask Lumo to get the Node.js process.argv for us. That is,

(def args (.slice (.-argv process) 3))

is the Lumo version of Node’s

var args = process.argv.slice(3)

We’re ignoring the first few arguments because they are our script’s name and lumo itself and whatnot.

$ time ./hello-lumo
Hello, World!

real    0m1.504s
user    0m1.840s
sys     0m0.148s

That’s a little faster, but still not great. Plus we pretty much still have to know Node.js to write it.

Summary

So, they’re all pretty slow.

And it would be nice if the Clojure code didn’t have to change. Like if there were some Clojure way to get the command line arguments and then Planck and Lumo made it look more like that. Interoperability with WebKit and Node is nice, but it would be even nicer if it were more transparent.

Command line Clojure

Rust and QML on Xubuntu

I just spotted A simple Rust GUI with QML on r/rust and wanted to give it a try. I already had Rust installed, but not QML. My desktop computer is currently running Xubuntu, which is Ubuntu with Xfce. Xfce is based on GTK+, whereas QML is based on Qt. There is a Kubuntu, which is Ubuntu with KDE. Since KDE is based on Qt, it probably has all the stuff we need for QML already. But we needn’t switch to Kubuntu to install QML. We can install it on Xubuntu or any version of Ubuntu just fine 1.

To install QML, I ran

sudo apt install qtdeclarative5-dev qml-module-qtquick-controls

which installed a whole bunch of dependencies, since I had no Qt things at all on this box yet. This got “Hello, World” to work, but there was not enough whitespace.

Screen shot of "hello world" in QML

Figure 1: “hello world” in QML!

I added width and height attributes to the .qml file and changed anchors.fill to anchors.centerIn

ApplicationWindow {
    visible: true
    width: 200
    height: 100
    Text {
        anchors.centerIn: parent
        text: message
    }
}

This looked a little better.

Screen shot of "hello world" with whitespace added

Figure 2: “hello world” with whitespace added

I probably should have made the font bigger as well, but I didn’t want to go too far down the “pointy-clicky nonsense” rabbit hole.

Later in the exercise, I needed QtQuick.Dialogs as well. For that, I ran

sudo apt install qml-module-qtquick-dialogs

Anyway, it’s a fun little blog post…you should give it a try! Bottom line is, Ubuntu has everything we need to follow along with it, even if the names aren’t that obvious.

Footnotes:

1

Indeed, renaming the distribution after the window manager like this is my least favorite thing about Ubuntu. I appreciate easy access to Xfce, but I didn’t need a whole new name. Xubuntu, Kubuntu, Lubuntu,…gah!

Rust and QML on Xubuntu

Strings in Go and Rust

This week at Go Meetup, we talked briefly about how strings in Go are UTF-8, but not really. What I mean is, on the one hand, we can write

s := "Hello, 世界!"
fmt.Println(s)

and it prints out

Hello, 世界!

as expected. But on the other hand, we can put an invalid UTF-8 sequence into a string as well

s := "\x67\x72\xfc\xdf\x65"

It will compile just fine, but print out junk.

gr��e

If we accept strings from an external source, we probably don’t want to do stringy things with them without first checking that they’re valid. For example, this code

package main

import (
    "fmt"
    "os"
)

func main() {
    for _, s := range os.Args {
        fmt.Println(s)
    }
}

just prints whatever we give it

$ ./garbage foo bär $(echo -en "\x67\x72\xfc\xdf\x65") baz
./garbage
foo
bär
gr��e
baz

while this one

package main

import (
    "fmt"
    "os"
    "unicode/utf8"
)

func main() {
    for _, s := range os.Args {
        if utf8.ValidString(s) {
            fmt.Println(s)
        } else {
            fmt.Println("not valid")
        }
    }
}

only prints valid strings

$ go build valid_string.go 
$ ./valid_string foo bär $(echo -en "\x67\x72\xfc\xdf\x65") baz
./valid_string
foo
bär
not valid
baz

In Rust, strings are UTF-8 as well. We can write

let s = "Hello, 世界!";
println!("{}", s);

and it prints out

Hello, 世界!

as expected. But unlike Go, we can’t put an invalid UTF-8 sequence in a string. This

let s = "\x67\x72\xfc\xdf\x65";

doesn’t even compile

error: this form of character escape may only be used with characters in the range [\x00-\x7f]

However, we still need to be careful. This

let v = vec![0x67, 0x72, 0xfc, 0xdf, 0x65];
let t = String::from_utf8(v);
println!("{:?}", t);

compiles fine, but gives a run-time error

Err(FromUtf8Error { bytes: [103, 114, 252, 223, 101], error: Utf8Error { valid_up_to: 2 } })

So once again, if we accept strings from an external source, we probably don’t want to do stringy things with them without first checking that they’re valid. But, unlike in Go, we can’t even put them in a string until we check. This code

use std::env;

fn main() {
    for arg in env::args() {
        println!("{}", arg);
    }
}

panics if any arguments are not valid UTF-8

$ ./valid_string_panic foo bär $(echo -en "\x67\x72\xfc\xdf\x65") baz
./valid_string_panic
foo
bär
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: "gr��e"', ../src/libcore/result.rs:837
note: Run with `RUST_BACKTRACE=1` for a backtrace.

Instead of std::env::args, we can use std::env::args_os to collect the arguments

use std::env;

fn main() {
    for arg in env::args_os() {
        println!("{:?}", arg);

        //println!("{}", arg);
        // does not compile
    }
}

This gives us an OsString instead of a String. Right away, we can see it’s different because it won’t even compile if we try to print it with “{}”. When we change to “{:?}”, we get junk for invalid UTF-8

$ ./valid_string_garbage foo bär $(echo -en "\x67\x72\xfc\xdf\x65") baz
"./valid_string_garbage"
"foo"
"bär"
"gr��e"
"baz"

To check that it’s valid, we can try to convert the OsString to a String. The to_str method returns an Option, which we can check

use std::env;

fn main() {
    for arg in env::args_os() {
        match arg.to_str() {
            Some(s) => println!("{}", s),
            None => println!("not valid"),
        }
    }
}

Thus we get

$ rustc valid_string.rs
$ ./valid_string foo bär $(echo -en "\x67\x72\xfc\xdf\x65") baz
./valid_string
foo
bär
not valid
baz

just as in Go.

So even though both Go and Rust use UTF-8 for strings, they are not the same model. There’s more to it. When it comes to encodings, there’s always more to it!

Strings in Go and Rust

Rusty Perl

I wanted to call Rust from Perl, so I tried to follow along with this blog post which does exactly that. But it was written before the release of Rust 1.0, so not everything still works. Here’s what I did:

Create a new Rust project called points.

$ cargo new points
     Created library `points` project
$ cd points

Add a lib section to the Cargo.toml file to create a .so instead of a .rlib.

[package]
name = "points"
version = "0.1.0"
authors = ["oylenshpeegul <oylenshpeegul@gmail.com>"]

[lib]
name = "points"
crate-type = ["dylib"]

Now edit the src/lib.rs as @pauldwoolcock describes, but there’s no deriving, no box, no int, and abs_sub is deprecated.

#[derive(Copy, Clone)]
pub struct Point { x: i64, y: i64 }

struct Line { p1: Point, p2: Point }

impl Line {
    pub fn length(&self) -> f64 {
        let xdiff = self.p1.x - self.p2.x;
        let ydiff = self.p1.y - self.p2.y;
        ((xdiff.pow(2) + ydiff.pow(2)) as f64).sqrt() 
    }
}

#[no_mangle]
pub extern "C" fn make_point(x: i64, y: i64) -> Box<Point> {
    Box::new( Point { x: x, y: y } )
}

#[no_mangle]
pub extern "C" fn get_distance(p1: &Point, p2: &Point) -> f64 {
    Line { p1: *p1, p2: *p2 }.length()
}

#[cfg(test)]
mod tests {
    use super::{Point, get_distance};

    #[test]
    fn test_get_distance() {
        let p1 = Point { x: 2, y: 2 };
        let p2 = Point { x: 4, y: 4 };
        assert!((get_distance(&p1, &p2) - 2.828427).abs() < 0.01f64);
    }
}

Now try running the tests!

$ cargo test
    Finished debug [unoptimized + debuginfo] target(s) in 0.0 secs
     Running target/debug/deps/points-0a1a2813ecad97ba

running 1 test
test tests::test_get_distance ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured

If we do a cargo build, we’ll get a debug build and our libpoints.so will be in one location

$ cargo build
    Finished debug [unoptimized + debuginfo] target(s) in 0.0 secs

If we do a cargo build --release, we’ll get a release build and our libpoints.so will be in another location.

$ cargo build --release
    Finished release [optimized] target(s) in 0.0 secs

To use libpoints.so from Perl, we’ll create a perl directory with our points.pl script in it.

$ mkdir perl
$ touch perl/points.pl

We’ll use the FindBin module to link to the libpoints.so file relative to where we are now. And we’ll use a debug flag to link to either the debug or the release.

#!/usr/bin/env perl

use v5.24;
use warnings;
use FindBin;
use FFI::Raw;

my $debug = shift;

my $libpoints = "$FindBin::Bin/../target/release/libpoints.so";
if ($debug) {
    $libpoints = "$FindBin::Bin/../target/debug/deps/libpoints.so";
}

my $make_point = FFI::Raw->new(
    $libpoints,
    'make_point',
    FFI::Raw::ptr,
    FFI::Raw::int, FFI::Raw::int,
);

my $get_distance = FFI::Raw->new(
    $libpoints,
    'get_distance',
    FFI::Raw::double,
    FFI::Raw::ptr, FFI::Raw::ptr,
);

my $p1 = $make_point->call(2,2);
my $p2 = $make_point->call(4,4);

my $result = $get_distance->call($p1, $p2);
say "The distance from (2,2) to (4,4) is $result (the square root of 8).";

Now we should be able to run the Perl script from anywhere, with either the debug build or the release build.

$ perl/points.pl 1
The distance from (2,2) to (4,4) is 2.82842712474619 (the square root of 8).

$ perl/points.pl 
The distance from (2,2) to (4,4) is 2.82842712474619 (the square root of 8).

All of these files are on github.

Rusty Perl

Epoch fail

I was initially excited when Elixir 1.3 was released with a Calendar module, but now that I’ve tried to use it in a project, I’m disappointed. It turns out that they didn’t include everything we need, so we still need to import the :calendar module from Hex.

Worse, there is now twice as much documentation to sift through to try to find the functions we need. Are we looking for DateTime.add or Calendar.DateTime.add? While I was working, I ended up keeping tabs open to not only Date, Time, DateTime, and NaiveDateTime, but also Calendar.Date, Calendar.Time, Calendar.DateTime, and Calendar.NaiveDateTime. I’m not certain, but it feels as though I’m actually worse off than I was without the 1.3 additions.

Adding to my frustration was that the module felt both overly pedantic and incomplete. I had to define my own function to find the date n months from a given date

defp plus_months(date, 0) do
  date
end
defp plus_months(date, n) when n > 0 do
  dim = Calendar.Date.number_of_days_in_month(date)
  plus_months(Calendar.Date.add!(date, dim), n-1)
end

That was to do this calculation where we add days and months and seconds.

  def google_calendar(n) do

    {whole_days, seconds} = div_rem(n, @seconds_per_day)
    {months, days} = div_rem(whole_days, 32)
    {:ok, date} = Date.new(1969,12,31)
    {:ok, time} = Time.new(0,0,0, {0,6})

    date = date
    |> Calendar.Date.add!(days)
    |> plus_months(months)

    Calendar.NaiveDateTime.from_date_and_time!(date, time)
    |> Calendar.NaiveDateTime.add!(seconds)

end

But I don’t actually care about a Date or Time; I just wanted to do all that to a NaiveDateTime. I think it’s overly pedantic to say we must add days to a Date rather than a NaiveDateTime. It doesn’t seem to buy us anything. Obviously we can do the whole calculation without choosing a time zone, so why all the hand-wringing?

What I really want to do is define a NaiveDateTime and add days, months, and seconds to it

def google_calendar(n) do

  {whole_days, seconds} = div_rem(n, 24 * 60 * 60)

  # A "Google month" has 32 days!
  {months, days} = div_rem(whole_days, 32)

  # A "Google epoch" is one day early.
  {:ok, datetime} = NaiveDateTime.new(1969,12,31,0,0,0,0)

  datetime
  |> plus_days(days)
  |> plus_months(months)
  |> plus_seconds(seconds)

end

But to do that I had to keep unwrapping and re-wrapping the NaiveDateTime, so it’s not worth it.

def plus_days(ndt, n) do
  d = NaiveDateTime.to_date(ndt) 
  t = NaiveDateTime.to_time(ndt) 
  {:ok, ndt} = NaiveDateTime.new(Calendar.Date.add!(d, n), t)
  ndt
end

def plus_months(ndt, 0) do
  ndt
end
def plus_months(ndt, n) do
  d = NaiveDateTime.to_date(ndt)
  dim = Calendar.Date.number_of_days_in_month(d)
  plus_months(plus_days(ndt, dim), n-1)
end

def plus_seconds(ndt, n) do
  Calendar.NaiveDateTime.add!(ndt, n)
end

This is the kind of gorgeous code I expect from Elixir

datetime
|> plus_days(days)
|> plus_months(months)
|> plus_seconds(seconds)

but to get it we have to do something stupid.

Note that the corresponding calculation in the Perl version of the same project is that gorgeous

Time::Moment
    ->from_epoch(-$SECONDS_PER_DAY)
    ->plus_days($days)
    ->plus_months($months)
    ->plus_seconds($seconds);

Perl’s Time::Moment gives us a single immutable object representing a date and time of day with an offset from UTC in the ISO 8601 calendar system. All of the methods above were supplied.

I hope we get something more like that in Elixir 1.4 or 1.5; the half-baked support we got in 1.3 does not seem helpful. I made a branch with :calendar removed to illustrate. It has the linear transformations in one direction, but not the other. It has only one of the other transformations.

Epoch fail