Thoughts of no consequence

Sampled Surveys

A fairly large part of my statistics lecture today was spent, directly and indirectly, on just how bad postal surveys are. Earnest letters in the mail imploring the recipient to take some time out of their day to fill in some boxes and then post them back (free of charge in an already-addressed envelope provided) get poor enough response rates when they decide national referendums or local elections—surveys from random organisations addressed “To the Householder” can only be expected to perform worse.

Stopping as you do in the toilet after class I was then confronted with a very common sight in university campus bathrooms: an A4 poster searching for volunteers for a student’s psychology project, or in this case criminology. The subject turned out to be experiences of pornography, which caused me to raise my eybrows a little bit, although not quite as much as the list of requirements, which so far as I recall were:

  • To identify as male or female
  • To have an opinion on whether pornography has positively or negatively affected them
  • To be heterosexual

I’m not sure if they meant the first to be “binary people only” (because they don’t think they will have enough enbys to properly analyse) or “everyone including both men and women but I think that this is the progressive way to say it” (in which case, more eyebrows, and that’s all I’m saying). The last one though seemed a bit strange: do you really want to exclude everyone not strictly heterosexual? The remainder seems important, especially in the generation likely to show up to a study advertised in a uni loo.

Though that being said, after other discussions in stats class lately the thing that really struck out at me was the middle one. Middle things are important, particularly people. Explicitly requesting people to show up who have “an opinion” is not a great way to get representation of people in between, and those people are, again, important to getting a proper picture to what people in general feel and have experienced. If this study winds up with only people who are strongly for or against its topic then it may have great difficulty figuring out why those people would have come to those positions.

So while I wish the student all the best in their endeavours you may have to excuse me if I am a little sceptical that their efforts—which will doubtless be quite significant in planning and carrying out this study—will actually produce anything reliable.

Projects

Calibre Recipes—News on your Kindle

The news icon from calibre

Calibre, a free and open source program for working with ebooks and ereaders, has some interesting capabilities when you poke around. One of these is the big red “Fetch News” button and dropdown.

Don’t ask me how I first found it—it was a while ago now, and anyway it is rather big and red. When you open ‘Schedule News Download’ in the dropdown menu you get a whole list of languages and places (e.g. “English,” “English (Ireland),” and “Irish”) and associated pre-built recipes, and selecting one and clicking “download now” begins what is potentially a rather protracted process of downloading and processing the RSS feeds into an ebook—albeit a rather special one.

An article in the calibre reader
An article from the io9 feed in the Calibre reader.

It doesn’t look like much in Calibre’s own viewer, but if sent—via email or over usb, both of which Calibre can help you with—to a kindle (and perhaps other ereaders) it creates a display of feeds and articles that can be leisurely browsed through.

But if you instead choose to add a custom news source you get the option to add your own feeds, building the magazine or newspaper you want to read in the morning. Further, options to schedule downloads mean that at specified times (provided your computer and Calibre are running) the feeds will be downloaded automatically.

You can see why this might be interesting, I hope.

Of course, that’s a big caveat—if you had to leave your computer on all night running Calibre so that it could prepare your news for the morning it would be a tad inconvenient. Luckily, if—big if, I know—you happen to have a server lying around (which is to say, a computer that is left on all night but is generally not on your desk in your bedroom lighting the whole room up while you’re trying to sleep) Calibre provides command-line tools that can be left to their own devices remotely.

Under the hood Calibre recipes appear to be python scripts. For example, you might have a file Religion.recipe with instructions to download the last two days of entries from the blogs of Kimberly Knight, Rachel Held Evans, Fred Clark, James McGrath, Libby Anne, and Jonny Scaramanga (up to a maximum of 100 each) that looks like the following:

class BasicUserRecipe1448244083(AutomaticNewsRecipe):
    title                 = u'Religion'
    oldest_article        = 2
    max_articles_per_feed = 100
    auto_cleanup          = True

    feeds = [
             (u'Kimberly Knight', u'http://feeds.feedburner.com/KimberlyKnight?format=xml'),
             (u'Rachel Held Evans', u"http://feeds.feedburner.com/RachelHeldEvans?format=xml"),
             (u'Slacktivist', u'http://feeds.feedburner.com/patheos/patheosslacktivist?format=xml'),
             (u'Explorting Our Matrix', u'http://feeds.feedburner.com/patheosmatrix?format=xml'),
             (u'Love, Joy, Feminism', u'http://feeds.feedburner.com/patheos/yiPT?format=xml'),
             (u'Leaving Fundimentalism', u'http://feeds.feedburner.com/LeavingFundamentalism?format=xml')
            ]

This can then be used directly from the command line with:

ebook-convert "Religion.recipe" "Religion.mobi" --tags="News,Religion" --authors="Various" --title="Religion" --series="Religion RSS"

This only then needs to be sent to your kindle, which can be done by a number of methods including Calibre’s own calibre-smtp command, like so:

calibre-smtp --relay="smtp.gmail.com [for example]" --port=587 --username="[your@email or similar username]" --password="[Your.Password]" --subject="News delivery - Religion.mobi" -a "Religion.mobi" [Your@email] [your@kindle.com] "News delivery (attached); Religion.mobi" >> log.txt 2>&1

This may take some setting up, both letting Calibre use your email and allowing your kindle to receive it, but you can find that information elsewhere.

After you’ve got that to work you can put it together in a script, allowing any number of different such recipes to be called by much shorter commands, e.g.:

#!/bin/bash
TODAY=$(date +"%F")
cd ~/calibre-recipies

rm "$1.mobi"
echo "Starting $1 at $(date)" >> log.txt
ebook-convert "$2.recipe" "$1.mobi" --pubdate=$TODAY --tags="News" --authors="Various" --title="$1" --series="$1 RSS" >> log.txt 2>&1
calibre-smtp --relay="[server]" --port=587 --username="[username]" --password="[password]" --subject="News delivery - $1.mobi" -a "$1.mobi" [email@address] [email@kindle.com] "News delivery (attached); $1.mobi" >> log.txt 2>&1
echo "End $1 at $(date)" >> log.txt

This could then be called with

# First "Religion" is the output name,
# second is the script name
./recipescript.sh "Religion" "Religion"

And that can be called at, say, 5am every weekday morning via the marvel that is cron with e.g.

20 5    * * Mon-Fri        /home/petra/calibre-recipes/recipescript.sh "Religion" "Religion"

There are, however, a few disadvantages: Not everything downloads properly all the time, and you often get random encoding errors. This is annoying, although I find you get used to it. It’s a great way to get you something to read on the train without needing internet as-you-read-it (just at home to download things) but not so great at keeping up with everything.

Still, you might be able to get it working for you, and there’s definitely a great deal that can be done to tweak it. For more information start with the documentation for the ebook-convert command.

Projects

Random Skylines

I play Cities: Skylines quite a bit, albeit not much in the last few weeks. I’m very excited just for the free features of the Mass Transit expansion but personally think the game was pretty great as it was more or less at launch—if we had been stuck with that without further updates it still would have been my “Game of the Year, Every Year.”

Cities Skylines Screenshot
Cities Skylines

Sometime last year I got into making my own maps to build on, a time-consuming but rewarding experience. The maps that the game comes with become limiting after a while, and although Steam has innumerable user-created maps floating around I prefer to shape my own. The problem is that I always feel my maps lack a certain something in the way of terrain features, particularly when it comes to things like the ridgelines and valleys I see so much around me where I live.

What I need, I think, is a random map generator that can create the kind of height map that I need for a given idea, and then I can tweak it as I want. The Skylines map editor already has 1081×1081 pixel greyscale heightmap import (and export) capabilities; all I need is to make an external tool to produce them.

As an aside, there already exists a tool called terrain.party which allows you to grab Skylines-compatible heightmaps from real terrain, but these have a number of flaws (particularly when it comes to the shoreline) and in my personal opinion carry too much attachment to what is really at that location in the real world.

Napier Height Map from terrain.party
Napier Height map from terrain.party

So this comes to the issue of random terrain generation, for which not all of the tutorials on the internet have “now finish the rest of the owl” syndrome (though too many do). After a bit of playing I rejected using the midpoint displacement algorithm as it relies too greatly on powers of 2 plus 1 (2^n + 1)—a set which 1081 definitely is not among. Instead I turned to Perlin noise, which is arguably cheating.

To summarise, when you typically assign random values to points you wind up with the value of one point being completely independent of the value of any of its neighbours. This would produce a useless, spiky terrain. Instead the value of a noise function can be expected to be relatively similar at nearby points, though still “random” over all. This would instead produce a landscape of rolling hills, all random and yet all the same. Perlin noise works by adding a whole lot of noise functions of different sizes together, producing a much more interesting if not quite realistic map.

A slight modification can be made to this to make mountains steeper and plains flatter, by raising a number (here 6) to the power of the perlin function.

#include <png++/png.hpp>
#include <iostream>
#include <libnoise/noise.h> // From the libnoise library
#include <ctime>
#include <utility>
#include <cmath> // Include libm for pow function

// ...

std::pair<double, double> perlinize(double ** map, int width, int height, int seed, double frequency, double persistance, double exponent) {
    noise::module::Perlin pgen; // From libnoise
    pgen.SetSeed(seed);
    pgen.SetFrequency(frequency);
    pgen.SetPersistence(persistance);
    double minvalue = 100.0, maxvalue = -100.0;
    for (int i = 0; i < width; i++) {
        for (int j = 0; j < height; j++) {
            double fnew = pow(exponent, pgen.GetValue(i, j, 0));
            minvalue = (fnew < minvalue) ? fnew : minvalue;
            maxvalue = (fnew > maxvalue) ? fnew : maxvalue;
            map[i][j] = fnew;
        }
    }
    return { minvalue, maxvalue }; // Return the maximum and minimum generated value
}
A perlin-derived heightmap
A Perlin-derived heightmap

But it would be nice if a bit of erosion could be applied to this map, particularly if the results could be used as a guide for placing rivers—a crucial part of Skylines, which attempts to model water movement including inflows into pumping stations and sewage outflow back into the environment. But I’m rather yet to get that to work…

Currently Reading

Currently Reading

For the last few days or so I have been reading Shadow and Claw by Gene Wolfe. It’s apparently an omnibus edition of the first two books in a series, which makes sense—I’m past the middle and into the Claw.

I put it in Science Fiction on my Kindle but keep forgetting that it’s not Fantasy. It’s got swords and torturers (or a sword and a torturer, in the second book), but the walls of the guild oubliette—which doesn’t particularly resemble the mechanic from Crusader Kings II—are made of metal and characters speak casually of humanity once exploring the stars but are presently relegated to a certain cold world around a dying star.

The book is very dark at times, what with the main character being, again, a torturer. There’s plenty else that I wont spoil, for that matter, but it isn’t all like that.

I am very fond of the writing style, especially after recently giving up on a book largely for that reason. It’s important! Especially when all I can think while I’m reading is “I could do this better,” which believe me is no complement. The digressions, and particularly the digressions accompanied by the narrator insisting that they’re completely irrelevant but you’re going to have to put up with them because it’s his account, are often the best parts. It flows; it’s enjoyable; I’m always reluctant to put it down when the train pulls into the station. My biggest caveat is really that I hope nobody is reading over my shoulder…

The book I would really rather be reading, however, is Ian McDonald’s Luna: Wolf Moon, the unfortunately titled (and also delayed) sequel to Luna: New Moon; “Game of Thrones in Space” according to some but “Hard Science Fiction Dune” to me. I say unfortunately titled because you would expect a completely different genre from what it actually is—at least for the most part! Alas, while it finally appears to be out I cannot afford it right now. Instead I’m stuck with humble bundles and ebook givaways, but the moral of the story is that those aren’t all bad.