Julia in Jupyter

screen shot of Julia Jupyter notebook

Last month, Julia 1.0 was released– hooray! I installed it on my Ubuntu 18.04 machine with

cd ~/Downloads
gpg --import juliareleases.asc
gpg --verify julia-1.0.0-linux-x86_64.tar.gz.asc

cd ~/julia
tar xf ~/Downloads/julia-1.0.0-linux-x86_64.tar.gz

cd ~/bin
ln -s ../julia/julia-1.0.0/bin/julia

That last part linked it to by bin directory, which is already in my path.

$ which julia
/home/tim/bin/julia

$ julia --version
julia version 1.0.0

Today, I tried to use Julia from a Jupyter notebook. Jupyter is already installed.

$ which jupyter
/usr/bin/jupyter

$ jupyter --version
4.4.0

I installed it last year with

sudo apt install jupyter-core jupyter-notebook

This has been working fine and I have notebooks in Python 2, Python 3, Perl 5, Perl 6, and Clojure. However, Julia didn’t like it.

$ julia
julia> using Pkg
julia> Pkg.add("IJulia")
...
julia> using IJulia
[ Info: Precompiling IJulia [7073ff75-c697-5162-941a-fcdaad2a7d2a]
ERROR: LoadError: IJulia not properly installed. Please run Pkg.build("IJulia")
Stacktrace:
...

Running Pkg.build and a little bit of googling revealed that I didn’t have a jupyter-kernelspec command. Why not?

$ jupyter-kernelspec

Command 'jupyter-kernelspec' not found, but can be installed with:

sudo apt install jupyter-client

Ah. I hadn’t installed jupyter-client, just jupyter-core and jupyter-notebook. Running that command and then rebuilding IJulia worked a treat. Now, either

julia> using IJulia
julia> notebook()

from the Julia prompt, or

$ jupyter notebook

from the shell prompt starts up Jupyter (and Julia 1.0 is now on the list of kernels from which to choose). I prefer the latter, since I change to my Jupyter directory first, which shows all of my notebooks. Starting it within Julia shows my entire home directory.

Anyway, the bottom line is that the Julia 1.0 executable seems to work great with the Jupyter provided by Ubuntu; we just have to install the client too!

sudo apt install jupyter-core jupyter-notebook jupyter-client
Advertisements
Julia in Jupyter

Breaking down

I just came across this talk on category theory by Eugenia Cheng. I thought the whole thing was pretty great, but I particulary liked this impromptu remark about the nature of mathematics:

A lot of maths is about how we can take individual objects and build them into new ones that are a bit more interesting. The converse of that is that if we can then understand big ones, relative to small ones, and break them down into small ones, then we don’t have to do as much work. Because then all we have to do is understand some small objects and building processes. And then we can understand anything. That’s how all of maths works. That’s why we like understanding prime factorisation, so that we can just understand prime numbers and then build them into big numbers. And why we like breaking topological spaces into small pieces and then building them all up out of circles and disks. And why we like expressing all groups as fundamental finite ones and then seeing how we can build everything else with that. It’s so that we don’t ever have to deal with complicated things. We can just think about simple ones and building processes.

It reminds me of what Rich Hickey said about design:

So what do I mean when I say good design? I think one of the most interesting things about design is that, you know, people think it’s generating this intricate plan. But designing is not that. Designing is fundamentally about taking things apart. It’s about taking things apart in such a way that they can be put back together. If that makes sense. So separating things into things that can be composed. That’s what design is. Right? If you just make this intricate thing, you really haven’t designed in a way that needs to support the things that designs need to support. Like change. Every component of a design should be kind of about one or very few things. That’s sort of the nature of it, breaking things down until they are nearly atomic. And only then do you take those things that you have broken apart and compose them to solve the problem you set out to solve. But the first job is to take things apart in such a way that you can do that. And a good design process is iterative. This isn’t a grand plan you do once and then you go and finish.

Perhaps software design isn’t as far from category theory as Hickey seems to think!

Breaking down

Programming Rust

cover of Programming Rust book

I’m currently reading Programming Rust by Jim Blandy and Jason Orendorff. I’m really enjoying it.

I bought it after hearing this talk by Orendorff. I loved the talk, so I bought the book. I figured if he can put together a talk that good, he can probably put together a pretty good book too. And I was right!

More recently, I listened to this podcast with Blandy. I really enjoyed this as well. So it’s no wonder I’m enjoying the book. I have no idea how they divided the writing duties, but it doesn’t matter— I like both styles.

Programming Rust

Autopsy 4.7 in Linux

Last year, I got Autopsy 4.5 to work in Linux, including processing some data. Now, Autopsy 4.7 is out and it claims even better Linux support. Today, I decided to try it.

First, I’ll uninstall the existing sleuthkit.

$ cd ~/sleuthkit/sleuthkit-sleuthkit-4.5.0
$ sudo make uninstall
...

Now, I’ll follow the instructions for Linux

$ sudo apt install libvhdi1 libvmdk1 libvhdi-dev libvmdk-dev libpostgresql-jdbc-java libc3p0-java
...
$ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
$ sudo dpkg -i ~/Downloads/sleuthkit-java_4.6.1-1_amd64.deb
...
$ cd ..
$ unzip -l ~/Downloads/autopsy-4.7.0.zip
$ cd autopsy-4.7.0/bin
$ ./autopsy 
$ chmod +x autopsy
$ ./autopsy 
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.netbeans.ProxyURLStreamHandlerFactory (file:/home/tim/Autopsy/autopsy-4.7.0/platform/lib/boot.jar) to field java.net.URL.handler
WARNING: Please consider reporting this to the maintainers of org.netbeans.ProxyURLStreamHandlerFactory
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Library not found in jar (libtsk_jni)
SleuthkitJNI: failed to load libtsk_jni

Hrm. It didn’t work. The graphical window says this

org.netbeans.InvalidException: StandardModule:org.sleuthkit.autopsy.core jarFile: /home/tim/Autopsy/autopsy-4.7.0/autopsy/modules/org-sleuthkit-autopsy-core.jar: java.lang.UnsatisfiedLinkError: org.sleuthkit.datamodel.SleuthkitJNI.getVersionNat()Ljava/lang/String;

So where is libtsk_jni?

$ dpkg -l | grep sleuth
ii  sleuthkit-java                          4.6.1-1                                    amd64        tools for forensics analysis on volume and filesystem data

$ dpkg --contents sleuthkit-java_4.6.1-1_amd64.deb
drwxr-xr-x root/root         0 2018-05-08 11:38 ./
drwxr-xr-x root/root         0 2018-05-08 11:38 ./usr/
drwxr-xr-x root/root         0 2018-05-08 11:38 ./usr/lib/
drwxr-xr-x root/root         0 2018-05-08 11:38 ./usr/lib/x86_64-linux-gnu/
-rw-r--r-- root/root    935224 2018-05-08 11:38 ./usr/lib/x86_64-linux-gnu/libtsk.so.13.4.2
-rw-r--r-- root/root     75042 2018-05-08 11:38 ./usr/lib/x86_64-linux-gnu/libtsk_jni.a
-rw-r--r-- root/root   1719320 2018-05-08 11:38 ./usr/lib/x86_64-linux-gnu/libtsk.a
-rw-r--r-- root/root    133480 2018-05-08 11:38 ./usr/lib/x86_64-linux-gnu/libtsk_jni.so.0.0.0
-rw-r--r-- root/root      1174 2018-05-08 11:38 ./usr/lib/x86_64-linux-gnu/libtsk.la
-rw-r--r-- root/root      1225 2018-05-08 11:38 ./usr/lib/x86_64-linux-gnu/libtsk_jni.la
drwxr-xr-x root/root         0 2018-05-08 11:38 ./usr/share/
drwxr-xr-x root/root         0 2018-05-08 11:38 ./usr/share/java/
-rw-r--r-- root/root   5131696 2018-05-08 11:38 ./usr/share/java/sqlite-jdbc-3.8.11.jar
-rw-r--r-- root/root   1399359 2018-05-08 11:38 ./usr/share/java/sleuthkit-4.6.1.jar
drwxr-xr-x root/root         0 2018-05-08 11:38 ./usr/share/doc/
drwxr-xr-x root/root         0 2018-05-08 11:38 ./usr/share/doc/sleuthkit-java/
-rw-r--r-- root/root       512 2018-05-08 11:32 ./usr/share/doc/sleuthkit-java/copyright
-rw-r--r-- root/root       196 2018-05-08 11:33 ./usr/share/doc/sleuthkit-java/changelog.Debian.gz
lrwxrwxrwx root/root         0 2018-05-08 11:38 ./usr/lib/x86_64-linux-gnu/libtsk.so.13 -> libtsk.so.13.4.2
lrwxrwxrwx root/root         0 2018-05-08 11:38 ./usr/lib/x86_64-linux-gnu/libtsk_jni.so -> libtsk_jni.so.0.0.0
lrwxrwxrwx root/root         0 2018-05-08 11:38 ./usr/lib/x86_64-linux-gnu/libtsk.so -> libtsk.so.13.4.2
lrwxrwxrwx root/root         0 2018-05-08 11:38 ./usr/lib/x86_64-linux-gnu/libtsk_jni.so.0 -> libtsk_jni.so.0.0.0

$ ls -l /usr/lib/x86_64-linux-gnu/libtsk*
-rw-r--r-- 1 root root 1719320 May  8 11:38 /usr/lib/x86_64-linux-gnu/libtsk.a
-rw-r--r-- 1 root root   75042 May  8 11:38 /usr/lib/x86_64-linux-gnu/libtsk_jni.a
-rw-r--r-- 1 root root    1225 May  8 11:38 /usr/lib/x86_64-linux-gnu/libtsk_jni.la
lrwxrwxrwx 1 root root      19 May  8 11:38 /usr/lib/x86_64-linux-gnu/libtsk_jni.so -> libtsk_jni.so.0.0.0
lrwxrwxrwx 1 root root      19 May  8 11:38 /usr/lib/x86_64-linux-gnu/libtsk_jni.so.0 -> libtsk_jni.so.0.0.0
-rw-r--r-- 1 root root  133480 May  8 11:38 /usr/lib/x86_64-linux-gnu/libtsk_jni.so.0.0.0
-rw-r--r-- 1 root root    1174 May  8 11:38 /usr/lib/x86_64-linux-gnu/libtsk.la
lrwxrwxrwx 1 root root      16 May  8 11:38 /usr/lib/x86_64-linux-gnu/libtsk.so -> libtsk.so.13.4.2
lrwxrwxrwx 1 root root      16 May  8 11:38 /usr/lib/x86_64-linux-gnu/libtsk.so.13 -> libtsk.so.13.4.2
-rw-r--r-- 1 root root  935224 May  8 11:38 /usr/lib/x86_64-linux-gnu/libtsk.so.13.4.2

That seems right…why can’t it find it? Oh, well; forget the package…

sudo dpkg -r sleuthkit-java
sudo dpkg --purge sleuthkit-java

Let’s try building the entire sleuthkit from source

$ cd ~/sleuthkit
$ tar xf ~/Downloads/sleuthkit-4.6.1.tar.gz
$ cd sleuthkit-4.6.1

# Hey, there's a configure script already!
$ ./configure
$ make
$ make check
$ sudo make install

# All set?
$ which mmls
/usr/local/bin/mmls
$ ls bindings/java/dist/
sleuthkit-4.6.1.jar

$ cd ~/sleuthkit
$ unzip ~/Downloads/autopsy-4.7.0.zip
$ cd autopsy-4.7.0/
$ bash unix_setup.sh
photorec found
Java found in /usr/lib/jvm/java-8-openjdk-amd64
/usr/local/share/java/sleuthkit-4.6.1.jar found
Copying into the Autopsy directory
Autopsy is now configured. You can execute bin/autopsy to start it

$ bin/autopsy
...
SleuthkitJNI: loaded libtsk_jni
...

Success! Aw, but when I started a new case, it crashed!

screen shot of creating case

The pop-up window said

Sorry, the application java has stopped unexpectedly.

If you notice further problems, try restarting the computer.

and the terminal said

# A fatal error has been detected by the Java Runtime Environment:
...
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P" (or dumping to /home/tim/core.21205)
#
# An error report file with more information is saved as:
# /home/tim/hs_err_pid21205.log
...
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
...

So. I don’t really know what to do about this setback. This Ubuntu machine is now at 18.04, but I doubt that’s the problem. I’m using OpenJDK, not Oracle Java…but if that’s the issue, then I don’t want to play.

Autopsy 4.7 in Linux

In Praise of Higher Order Functions

In February, Mary Sheeran gave a talk at LambdaDays called “In praise of Higher Order Functions and of some friends and heroes”. I was not able to attend LambdaDays, but I watched the video of the talk and I really enjoyed it.

screen shot of video playing

Sheeran is urging us to read these historic papers, but I was disappointed to be unable to find many of them on the internet. It wasn’t until her 41st slide that she referred to something that we can see unimpeded! I guess it’s not really possible for an amateur to read all of these papers. One really must belong to a university or some other institution that subscribes to these journals and has (or can borrow) these books. What a shame.

Sheeran ended her talk with an admonishment from Christopher Strachey to bring together theory and practice.

It has long been my personal view that the separation of practical and theoretical work is artificial and injurious. Much of the practical work done in computing, both in software and in hardware design, is unsound and clumsy because the people who do it have not any clear understanding of the fundamental design principles of their work. Most of the abstract mathematical and theoretical work is sterile because it has no point of contact with real computing.

Strachey said that over fifty years ago in another paper I can’t find on the internet (“Towards a formal semantics”, 1966), but it doesn’t seem like we’ve listened. I’m a working programmer and I can’t easily read this research.

Anyway, I did track down some of the papers, so I thought I’d post the links I found. Some of the links are PDFs.

John McCarthy

  • LISP Programmer’s Manual
    • Modification number 7 – March 3, 1959
    • Modification number 12 – March 20, 1959
    • I found one version (not sure which one) behind a paywall
  • LISP 1.5 Programmer’s Manual, second edition, fifteenth printing, 1985

Christopher Strachey

  • Strachey began experimenting with higher-order functions in 1961
  • Strachey developed R1 (foldr) and R2 (foldl)
  • David W. Barron reproduced the 1963 workshop from audio recordings
  • Barron and Strachey, “Programming”, 1966 (chapter three of Advances in Programming and Non-Numerical Computation, which you can buy from Elsevier)
  • Oliver Danvy and Michael Spivey, “On Barron and Strachey’s Cartesian Product Function”, 2007 (buy from the ACM).
  • Bitwise Operations, 1961 (buy from the ACM).

Kenneth Iverson

Graham Hutton

Guy E. Blelloch

Duane Merrill and Andrew Grimshaw

Horacio Gonzalez-Velez and Mario Leyton

  • A survey of algorithmic skeleton frameworks: high-level structured parallel programming enablers, 2010 (behind a paywall)
In Praise of Higher Order Functions

CPOSC 2017

Yesterday, I drove up to Lancaster for the Central Pennsylvania Open Source Conference. It was terrific! This is my third time doing this and it’s become one of my favorite conferences. There are three tracks of talks most of the day, so you can almost always find something interesting to learn about at every session. Often, you have to decide between two or more talks that you’d really like to see.

pic of empty stage

Don Schenck had lots of practical advice for beginners in a talk called “OMG How Do I Even Start An Open Source Project?”

Matt Rogish talked about “Kubernetes 101” and how to pronounce kubectl (“coob see tee el” or “coo bee cuttle”).

In “Kubernetes: A Deep Dive,” Mike Newswanger pronounced kubectl “cube control.”

Rogish and Newswanger both used Docker for all of their examples. They both mentioned that you could run Kubernetes without Docker, but said no more about it. I was kind of hoping to hear about using Kubernetes to orchestrate other things.

Jason O’Donnell talked about “Vault: A Tool for Managing Secrets” and had a nice demo.

James Tramel did an “Introduction to Blockchain.” This was nearly impossible to do in thirty minutes. He did a fine job, but there were clearly more questions in the room when we had to stop.

Sarah Mogin gave a really great talk called “Building for Performance and Scale with React/Redux.” I don’t usually have much patience for JavaScript, but she had ton of practical advice with no sugar coating.

“Man, JavaScript is annoying.” – @sarahmogin

Much of her talk had an A versus B structure for a variety of A’s and B’s (traditional versus headless, Component versus PureComponent, &c.) which was very effective.

pic of Sarah Mogin

I also really enjoyed Steve High‘s talk “Ditching REST for gRPC.” His demo was an arithmetic service “because who doesn’t like math.” He live-coded subtract into the service during the talk. He didn’t hear me answer “minuend” when he asked, so his variables were called firstnumber and subtrahend. Naming things is hard.

pic of Steve High

There were also lightning talks, 3D printing by PrintedSolid.com, and lots of great local food.

The folks from MajorMega had a prototype of Hyperland set up that we could try out. In addition to VR headset and controllers, it had fans for wind and a floor that tipped. This guy thinks he is riding on the back of a moving truck and shooting at other cars and things chasing him.

pic of Hyperland prototype

There were also giveaways, including a 3D Printer from LulzBot. I didn’t win anything, but that’s okay. It was fantastic day!

pic of organizers

CPOSC 2017