When working with datasets from PATHspider it’s very easy to max out your memory. When running an analysis recently, I found my computer locking up and generally being unresponsive. This new machine should be able to cope so this was odd, and then I realised that I was running two jupyter notebooks and the other had previously loaded all the data into memory, so this second one pushed me into swap. 6GB of memory got swapped and then the machine was near useless almost instantly.
Still in the process or tidying up the bot behind the @TorAtlas Twitter account, which means I’m still writing Java. I’ve now been exploring “list comprehension"-like techniques using another new Java 8 feature, Streams. Essentially this involves creating lists from lists in an elegant way.
In Python, I would do something like:
runningRelays = [relay for relay in relays if relay['is_running']]
(I know something is happening because writing that example line I had an incredible urge to terminate the line with a semicolon.)
I’m in the process or tidying up the bot behind
TorAtlas and I’ve chosen to write it in Java.
My undergraduate degree had a lot of Java in it, but that was while ago now,
and there are some cool new features I’ve learnt about including Period and
Duration.
Some of the tweets coming from the bot talk about how long a relay has been
contributing to the Tor network. When fetching the details
document I look at the
first_seen field and get this as a
ZonedDateTime
to make sure I’m doing all my calculations in UTC. I then use the between()
method on various ChronoUnits to work out how long between the first time the
relay was seen and now, and then I produced something human readable from this: