Hi!
This is the personal homepage of derf / derfnull. Every now and then, I dabble in software/hardware projects, take pictures, or write blogposts.
Projects
I'm a frequent user of public transport and enjoy building API clients and web services to provide as much transit data as I can find in a structured manner. You'll also find some energy measurement utilities and assorted minor hardware hacks here.
Public Transport Websites
Public Transport CLIs
> efa Essen Martinstr Düsseldorf Hbf 14:34 ab Essen Martinstr.: Bstg. 1 Straßenbahn 108 Essen Altenessen Bf Schleife 14:38 an Essen Hauptbahnhof: Bstg. 1 14:47 ab Essen Hauptbahnhof: 2 R-Bahn RE11 (RRX) Düsseldorf Hbf 15:24 an Düsseldorf Hbf: 10
> db-wagenreihung 'Essen Hbf' 723 ▏ G ▕▏ F ▕▏ E ▕▏ D ▕▏ C ▕▏ B ▕▏ A ▕ > 39 38 37 36 35 33 32 31 29 28 27 26 25 23 22 21 >
> hafas-m 'Messegelände, Leipzig' 14:32 STR 16 Lößnig, Leipzig 14:40 STR 16 Sportforum Süd, Leipzig 14:43 ALT 86 S-Bahnhof Messe, Leipzig 14:47 STR 16 Lößnig, Leipzig
> db-iris 'Dortmund Hbf' 14:38 +16 IC 2027 Passau Hbf 11 14:39 ABR RE11 Kassel-Wilhelmshöhe 8 14:41 RE 57 Winterberg(Westf) 2 └──── RE 57 Brilon Wald 2 14:41 S 5 Hagen Hbf 5 14:42 S 2 Dortmund Hbf 6 14:45 +1 RE 1 Aachen Hbf 16
> ura-m Talbot 14:49:41 52 Aachen Bushof 15:04:47 11 Lichtenbusch 15:05:00 52 Eschweiler Bushof 15:18:00 1 Aachen Bushof 15:19:56 11 Hoengen Markt 15:35:00 1 Schevenhütte
> efa-m Dortmund 'Universität S' 08:32 +1 02 445 Dortmund Am Kai 08:35 3 HB1 Dortmund Technologiezentrum 08:36 3 HB1 Dortmund Eichlinghofen H-Bahn 08:38 02 447 Dortmund Bandelstraße 08:39 2 S1 Dortmund Hbf 08:40 01 447 Dortmund Hacheney
Other CLI Software
> ct a mutt mutt: retrieving package Cloning into 'mutt'... [..] created .muttrc -> /home/derf/packages/mutt/etc/muttrc
> ekgping ccc.de __________^________^__________^________^______
Energy Measurements
Blinkenlights
Utilities
Attic
I am no longer working on these projects.
News
Travel::Status::DE::IRIS v1.69
Travel-Status-DE-IRIS-1.69.tar.gz (signature)
- Repair Travel::Status::DE::IRIS::Stations module (broken in Release 1.68)
Travel::Status::DE::IRIS v1.68
Travel-Status-DE-IRIS-1.68.tar.gz (signature)
- Update IRIS station list
Travel::Status::DE::IRIS v1.67
Travel-Status-DE-IRIS-1.67.tar.gz (signature)
IRIS->new_p
: Fix bug inrelated_stations
accessor- Update IRIS station list
Travel::Status::DE::IRIS v1.66
Travel-Status-DE-IRIS-1.66.tar.gz (signature)
- Treat Essen-Dellwig and Essen-Dellwig Ost as separate stations. Although they are grouped in the IRIS backend, they are distinct stations. Both have platforms 1 & 2, so grouping them leads to confusion.
- Update IRIS station list
Thoughts on embedded environmental sensors
Over the past few years, I've been frequently working with I²C environmental sensors for measuring temperature, humidity and so on. Here are some thoughts and observations of sensors and breakout boards I made along the way. Note that this is by no means a proper professional review, you should take everything posted here with a grain of salt.
Minimal drivers for all sensors listed here can be found in the multipass project.
Sensors and Datasheet specs
Sensor | Property | Resolution | Accuracy | Range |
---|---|---|---|---|
AM2320 | Temperature [°c] Humidity [%] |
0.1 0.1 |
±0.5 ±3 |
-40 .. 80 0 .. 99.9 |
BME280 | Temperature [°c] Humidity [%] Pressure [hPa] |
0.01 0.008 0.18 Pa |
±1 @ 0 .. 65 ±3 @ 20 .. 80 ±1 |
-40 .. 85 0 .. 100 300 .. 1100 |
BME680 | Temperature [°c] Humidity [%] Pressure [hPa] VOC [IAQ] |
0.01 0.008 0.18 Pa 1 |
±1 @ 0 .. 65 ±3 @ 20 .. 80 ±0.6 ±15% ±15 |
-40 .. 85 0 .. 100 300 .. 1100 0 .. ? |
CCS811 | TVOC [ppb] | 1 | ? | 0 .. 1187 |
HDC1080 | Temperature [°c] Humidity [%] |
0.1 0.1 |
±0.2 @ 5 .. 60 ±2 |
-40 .. 125 0 .. 100 |
LM75B | Temperature [°c] | 0.125 | ±2 / ±3 | -55 .. +125 |
Notes
AM2320
- I²C readout is a multi-step process with special timing requirements
- Reported humidity appears to be far too low on some devices
BME280
- max 3.6V; some breakout boards provide LDO and level shifters for 5V operation
- Supports both I²C and SPI; operating mode selected by CSB value
- The breakout boards I am aware of connect VCC to both VDD and VDDIO, making power sequencing with respect to CSB a tad difficult. On some of them, I had to power CSB before providing power to VCC to ensure that the chip starts up in I²C mode.
BME680
- IAQ calculation is only possible with a closed-source BLOB provided by Bosch SensorTec, setting that up on a Raspberry Pi is quite easy though.
CCS811
- In addition to Total Volatile Organic Compound (TVOC), the sensor reports “equivalent CO₂” (eCO₂) data calculated from TVOC. I found these to be unreliable.
LM75B
- readout is trivial
- SMBus compatible: using it on a Raspberry Pi is a simple as
i2cget -y 1 0x48 0x00 w
HDC1080
- reported humidity appears to be a tad too low
Handling USB-serial connection issues on some ESP8266 dev boards
Depending on the configuration of a few GPIO pins during reset, ESP8266 chips can boot into a variety of modes. The most common ones are flash startup (GPIO0 low → execute the program code on a flash chip connected to the ESP8266) and UART download (GPIO0 high → transfer program code from UART to the flash chip).
Most development boards use the serial DTR and RTS lines of their usb-serial converter to allow reset (and boot mode selection) of the ESP8266 by (de)assertion of the DTR/RTS signals. esptool also uses this method when uploading new firmware to the flash.
Usually, things just work™ and an ESP8266 can be used with esptool, nodemcu-uploader, miniterm/screen, and other software. If esptool/nodemcu-uploader work, but miniterm/screen do not (and show repeating gibberish or nothing at all instead), the reason may be unusual DTR/RTS behaviour. I found manual control of DTR/RTS to help in this case:
- Connect to the serial device
- de-assert DTR
- de-assert RTS
- receive a working UART connection
For example, in pyserial-miniterm these signals can be set on startup:
pyserial-miniterm --dtr 0 --rts 0 /dev/ttyUSB0 115200
They can also be toggled at runtime via Ctrl+T Ctrl+D (DTS) and Ctrl+T Ctrl+R (RTS).
$ pyserial-miniterm /dev/ttyUSB0 115200
--- Miniterm on /dev/ttyUSB0 115200,8,N,1 ---
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
--- DTR inactive ---
--- RTS inactive ---
Hello, World!
Vindriktning is a cheap USB-C powered particle sensor that uses three colored LEDs to indicate the amount of PM2.5 (i.e., particulate matter with a diameter of less than 2.5µm) as a proxy for indoor air quality. By default, it simply measures PM2.5 and indicates whether air quality is good, not so good, or poor – there is no digital read-out of PM2.5 values.
Luckily, adding an ESP8266 to integrate it with MQTT, HomeAssistant, InfluxDB, or other software is quite easy. However, while most examples use Arduino's C++ dialect for programming, I personally prefer to stick to the NodeMCU Lua firmware on ESP8266 boards. Here is my basic readout code for reference.
function uart_callback(data)
if string.byte(data, 1) ~= 0x16 or string.byte(data, 2) ~= 0x11 or string.byte(data, 3) ~= 0x0b then
print("invalid header")
return
end
checksum = 0
for i = 1, 20 do
checksum = (checksum + string.byte(data, i)) % 256
end
if checksum ~= 0 then
print("invalid checksum")
return
end
pm25 = string.byte(data, 6) * 256 + string.byte(data, 7)
print("pm25 = " .. pm25)
end
function setup_uart()
port = softuart.setup(9600, nil, 2)
port:on("data", 20, uart_callback)
end
setup_uart()
This code assumes that the Vindriktning's TX pin is connected to ESP8266 GPIO4 (labeled "D2" on most esp8266 devboards). As the ESP8266 only has a single RX channel, which we reserve for programming and debugging, I'm using a Software UART implementation. At 9600 baud, that's not an issue.
Travel::Status::DE::IRIS v1.65
Travel-Status-DE-IRIS-1.65.tar.gz (signature)
- Update IRIS station list
If you're running a MediaWiki 1.35 with PluggableAuth and LdapAuthentication2, there's two ways of supporting login for LDAP accounts and local accounts.
In LocalSettings.php, set either
$wgPluggableAuth_EnableLocalLogin = true;
or
$LDAPAuthentication2AllowLocalLogin = true;
They have slightly different UI, but work pretty much the same from a login perspective. However, the LDAPAuthentication2 variant does not support local account creation.
So, if you're getting an error message along the lines of "The supplied
credentials could not be used for account creation" when trying to register
a local account on your MediaWiki instance, you may need to set
$wgPluggableAuth_EnableLocalLogin = true;
in your LocalSettings.php.
Travel::Status::DE::IRIS v1.64
Travel-Status-DE-IRIS-1.64.tar.gz (signature)
- Add experimental
new_p
constructor for non-blocking operation with Mojo::Promise and Mojo::UserAgent.
Travel::Status::DE::IRIS v1.63
Travel-Status-DE-IRIS-1.63.tar.gz (signature)
- update station database
Travel::Status::DE::IRIS v1.62
Travel-Status-DE-IRIS-1.62.tar.gz (signature)
- import station name changes and removals from Winterfahrplan 2022
Measurement Automation with Korad Bench Supplies
A few years back, I bought an RND Lab RND 320-KA3005P bench power supply both for its capability of delivering up to 30V @ 5A, and for its USB serial control channel. The latter can be used to both read out voltage/current data and change all settings which are accessible from the front panel, including voltage and current limits.
This weekend, I finally got around to writing a proper Python tool for controlling and automating it: korad-logger works with most KAxxxxP power supplies, which are sold under brand names such as Korad or RND Lab.
Now, basic characteristics such as I-V curves are trivial to generate. For instance, here's the I-V curve for an unknown RGB power LED.
It's based on three calls of the following command.
bin/korad-logger --voltage-limit 5 --current-range '0 0.2 0.001' --save led-$color.log 210
At a sample rate of about 10 Hz and 1 mA / 10 mV resolution, the bench supply won't perform miracles. Nevertheless, it is quite handy. If you measure only current (e.g. in CV mode), or only voltage (CC mode), you can even get near 20 Hz.
Travel::Status::DE::IRIS v1.61
Travel-Status-DE-IRIS-1.61.tar.gz (signature)
- Switch to GIS::Distance(::Fast), as Geo::Distance is deprecated and Geo::Distance::XS is no longer available. Note that the Debian package still uses Geo::Distance, as GIS::Distance is not trivial to package for Debian.
Avoiding accidental bricks of MSP430FR launchpads
The MSP430FR launchpad series is a pretty nifty tool both for research and teaching. You get an ultra-low-power 16-bit microcontroller, persistent FRAM, and energy measurement capabilities, all for under $20.
Unfortunately, especially when it comes to teaching, there's one major drawback: Out of bound memory accesses which are off by several thousand bytes can permanently brick the CPU. This typically happens either due to a buffer overflow in FRAM or a stack pointer underflow (i.e., stack overflow) in SRAM.
This issue recently bit one of my students and it turns out that it could have been avoided. So I'll give a quick overview of symptoms, cause, and protection against it, both as a reference for myself and for others.
Symptoms
A bricked MSP430FR launchpad is no longer flashable or erasable via JTAG or BSL. Attempts to control it via MSP Flasher fail with error 16: "The Debug Interface to the device has been secured".
* -----/|-------------------------------------------------------------------- *
* / |__ *
* /_ / MSP Flasher v1.3.20 *
* | / *
* -----|/-------------------------------------------------------------------- *
*
* Evaluating triggers...
* Invalid argument for -i trigger. Default used (USB).
* Checking for available FET debuggers:
* Found USB FET @ ttyACM0 <- Selected
* Initializing interface @ ttyACM0...done
* Checking firmware compatibility:
* FET firmware is up to date.
* Reading FW version...done
* Setting VCC to 3000 mV...done
* Accessing device...
# Exit: 16
# ERROR: The Debug Interface to the device has been secured
* Starting target code execution...done
* Disconnecting from device...done
*
* ----------------------------------------------------------------------------
* Driver : closed (Internal error)
* ----------------------------------------------------------------------------
*/
Unless you know the exact memory pattern written by the buffer overflow (and it specifies a reasonable password length), there is no remedy I'm aware of. The CPU is permanently bricked.
Cause
MSP430FR CPUs use a unified memory architecture: Registers, volatile SRAM, and persistent FRAM are all part of the same address space. This includes fuses (“JTAG signatures”) used to secure the device by either disabling JTAG access altogether or protecting it with a user-defined password.
While write access to several CPU registers requires specific passwords and timing sequences to be observed, this is not the case for the JTAG signatures. Change them, reset the CPU, and it's game over.
The JTAG signatures reside next to the reset vector and interrupt vector at the
16-bit address boundary, within the address range from 0xff80
to 0xffff
. On
MSP430FR5994 CPUs, the (writable!) text segment ends at 0xff7f
and SRAM is
located in 0x1c00
to 0x3bff
. So, a small buffer overflow in a persistent
variable (located in FRAM) or a significant stack pointer underflow (starting in
SRAM, growing down, and wrapping from 0x0000
to 0xffff
) may overwrite the
JTAG signatures with arbitrary data.
Protection
MSP430FR CPUs contain a bare-bones Memory Protection Unit. It can partition the
address space into up to three distinct regions with 1kB granularity and
enforce RWX settings for each region. So, if we disallow writes to the 1kB
region from 0xfc00
to 0xffff
, we no longer have to worry about accidentally
overwriting the JTAG signatures. To do so, place the following lines in your
startup code:
MPUCTL0 = MPUPW;
MPUSEGB2 = 0x1000; // memory address 0x10000
MPUSEGB1 = 0x0fc0; // memory address 0x0fc00
MPUSAM &= ~MPUSEG2WE; // disallow writes
MPUSAM |= MPUSEG2VS; // reset CPU on violation
MPUCTL0 = MPUPW | MPUENA;
MPUCTL0_H = 0;
Note that this disallows writes not just to the JTAG signatures, but also to
part of the text segment as well as the interrupt vector table. If an
application dynamically alters interrupt vector table entries or uses
persistent FRAM variables at addresses beyond 0xfbff
, this method will break
the application. Most practical use cases shouldn't run into this issue.
Monitoring The Things Indoor Gateway (TTIG) via InfluxDB
The Things Indoor Gateway (TTIG) is an affordable LoRaWAN gateway, ideal for getting started with The Things Network or other setups. Here are two ways of monitoring its radio performance and feeding data into e.g. InfluxDB, so you can display the results in a small Grafana dashboard.
TTN Gateway Server API
The Things Stack's Gateway Server API allows requesting uplink and downlink stats of a gateway if you have an appropriate API key.
First, you need to navigate to the gateway page in your TTN console and create a new API key with “View gateway status” rights. Using this key and your gateway ID, you can request connection statistics:
> curl -H "Authorization: Bearer GATEWAY_KEY" \
https://eu1.cloud.thethings.network/api/v3/gs/gateways/GATEWAY_ID/connection/stats | jq
{
"last_uplink_received_at": "2021-09-12T11:00:41.490891018Z",
"uplink_count": "115",
"last_downlink_received_at": "2021-09-12T00:05:45.008438327Z",
"downlink_count": "2",
}
With a cronjob running every few minutes, you can pass the data to InfluxDB. I'm using the following Python script for this:
#!/usr/bin/env python3
# vim:tabstop=4 softtabstop=4 shiftwidth=4 textwidth=160 smarttab expandtab colorcolumn=160
import requests
def main(auth_token, gateway_id):
response = requests.get(
f"https://eu1.cloud.thethings.network/api/v3/gs/gateways/{gateway_id}/connection/stats",
headers={
"Authorization": "Bearer {auth_token}"
},
)
data = response.json()
uplink_count = data.get("uplink_count", 0)
downlink_count = data.get("downlink_count", 0)
requests.post(
"http://influxdb:8086/write?db=hosts",
f"ttn_gateway,name={gateway_id} uplink_count={uplink_count},downlink_count={downlink_count}",
)
if __name__ == "__main__":
main("GATEWAY_KEY", "GATEWAY_ID")
It's also possible to assign “Read gateway traffic” rights to an API key. I didn't play around with that yet.
USB-UART Logs
By soldering a 1kΩ resistor onto R86 on the TTIG PCB, you can enable its built-in CP2102N USB-UART converter. This allows you to use the USB port not just for power, but also for observing its debug output. See Xose Pérez' Hacking the TTI Indoor Gateway blog post for details.
With this hack, connecting the TTIG to a linux computer capable of sourcing up
to 900mA via USB will cause a /dev/ttyUSB
serial interface to apper. You can
use tools such as screen
or picocom
with a baud rate of 115200 to observe
the output. Apart from memory usage and time synchronization logs, it includes
a line similar to the following one for each received LoRa transmission:
RX 868.3MHz DR5 SF7/BW125 snr=9.0 rssi=-46 xtime=0x43000FB11517C3 - updf mhdr=40 DevAddr=01234567 FCtrl=00 FCnt=502 FOpts=[] 0151B4 mic=-1842874694 (15 bytes)
So you can log statistics about Received Signal Strength, Signal-to-Noise Ratio, Spreading Factor and similar.
The Python script I'm using for this is somewhat more involved:
#!/usr/bin/env python3
# vim:tabstop=4 softtabstop=4 shiftwidth=4 textwidth=160 smarttab expandtab colorcolumn=160
import re
import requests
import serial
import serial.threaded
import sys
import time
class SerialReader(serial.threaded.Protocol):
def __init__(self, callback):
self.callback = callback
self.recv_buf = ""
def __call__(self):
return self
def data_received(self, data):
try:
str_data = data.decode("UTF-8")
self.recv_buf += str_data
lines = self.recv_buf.split("\n")
if len(lines) > 1:
self.recv_buf = lines[-1]
for line in lines[:-1]:
self.callback(str.strip(line))
except UnicodeDecodeError:
pass
# sys.stderr.write('UART output contains garbage: {data}\n'.format(data = data))
class SerialMonitor:
def __init__(self, port: str, baud: int, callback):
self.ser = serial.serial_for_url(port, do_not_open=True)
self.ser.baudrate = baud
self.ser.parity = "N"
self.ser.rtscts = False
self.ser.xonxoff = False
try:
self.ser.open()
except serial.SerialException as e:
sys.stderr.write(
"Could not open serial port {}: {}\n".format(self.ser.name, e)
)
sys.exit(1)
self.reader = SerialReader(callback=callback)
self.worker = serial.threaded.ReaderThread(self.ser, self.reader)
self.worker.start()
def close(self):
self.worker.stop()
self.ser.close()
if __name__ == "__main__":
def parse_line(line):
match = re.search(
"RX ([0-9.]+)MHz DR([0-9]+) SF([0-9]+)/BW([0-9]+) snr=([0-9.-]+) rssi=([0-9-]+) .* DevAddr=([^ ]*)",
line,
)
if match:
requests.post(
"http://influxdb:8086/write?db=hosts",
data=f"ttn_rx,gateway=GATEWAY_ID,devaddr={match.group(7)} dr={match.group(2)},sf={match.group(3)},bw={match.group(4)},snr={match.group(5)},rssi={match.group(6)}",
)
monitor = SerialMonitor(
"/dev/ttyUSB0",
115200,
parse_line,
)
try:
while True:
time.sleep(60)
except KeyboardInterrupt:
monitor.close()
Travel::Status::DE::IRIS v1.60
Travel-Status-DE-IRIS-1.60.tar.gz (signature)
- Update IRIS station list; remove stations which no longer receive train services
EFA-APIs mit JSON nutzen
Die meisten deutschen Fahrplanauskünfte nutzen entweder EFA ("Elektronische FahrplanAuskunft") oder HAFAS ("HAcon Fahrplan-Auskunfts-System"). Die meisten EFA-Instanzen wiederum bringen mittlerweile native JSON-Unterstützung mit, so dass sie leicht von Skripten aus nutzbar sind. JSON-APIS wie die von https://vrrf.finalrewind.org sind damit weitgehend obsolet.
Hier ein Python-Beispiel für https://efa.vrr.de:
#!/usr/bin/env python3
import aiohttp
import asyncio
from datetime import datetime
import json
class EFA:
def __init__(self, url, proximity_search=False):
self.dm_url = url + "/XML_DM_REQUEST"
self.dm_post_data = {
"language": "de",
"mode": "direct",
"outputFormat": "JSON",
"type_dm": "stop",
"useProxFootSearch": "0",
"useRealtime": "1",
}
if proximity_search:
self.dm_post_data["useProxFootSearch"] = "1"
async def get_departures(self, place, name, ts):
self.dm_post_data.update(
{
"itdDateDay": ts.day,
"itdDateMonth": ts.month,
"itdDateYear": ts.year,
"itdTimeHour": ts.hour,
"itdTimeMinute": ts.minute,
"name_dm": name,
}
)
if place is None:
self.dm_post_data.pop("place_dm", None)
else:
self.dm_post_data.update({"place_dm": place})
departures = list()
async with aiohttp.ClientSession() as session:
async with session.post(self.dm_url, data=self.dm_post_data) as response:
# EFA may return JSON with a text/html Content-Type, which response.json() does not like.
departures = json.loads(await response.text())
return departures
async def main():
now = datetime.now()
departures = await EFA("https://efa.vrr.de/standard/").get_departures(
"Essen", "Hbf", now
)
print(json.dumps(departures))
if __name__ == "__main__":
asyncio.get_event_loop().run_until_complete(main())
Travel::Status::DE::IRIS v1.59
Travel-Status-DE-IRIS-1.59.tar.gz (signature)
- Stations:
get_station
,get_station_by_name
: Return immediately if requested station is undef or empty - Add stations Menningen-Leitishofen, Meßkirch, Sauldorf, Alvesta station, Lund Central, and Stockholm Central
Travel::Status::DE::IRIS v1.58
Travel-Status-DE-IRIS-1.58.tar.gz (signature)
Result->has_realtime
: Fix cancellations not being reported as realtimedb-iris
: Annotate arrival/departure times which should have realtime data, but don't, with a question mark.
Contact
Mail: derf@finalrewind.org
Jabber: derf@chaosdorf.de
IRC: derf0 @ oftc, hackint
PGP key 781BB707 1C6BF648 EAEB08A1 100D5BFB 5166E005
Elsewhere
Mastodon: @derf@social.antifa.gmbh
Twitter: @derfnull
GitHub: derf
Work: Embedded Software Systems @ Universität Osnabrück