petermolnar.net

Mi az az XMPP, miért jó, és hogyan használd?

Az azonnali üzenetküldők egyre gyalázatosabbak a felhasználói érdekek és adatvédelem terén, pedig jó ideje lenne lehetőség szabadon választani szolgáltatót, programot/felületet, de akár még futtathatnánk is saját rendszert.

Miért nem tudnak chat programok egymás között üzeneteket váltani?

Valószínű, hogy küldtél és/vagy fogadtál már emailt életedben, és teljesen általánosnak találtad, hogy (például) egy @gmail.com email címről minden gond nélkül lehet levelet küldeni egy @outlook.com, de akár egy @freemail.hu címre is, pedig ezeket a rendszereket egészen más cégek üzemeltetik.

Hasonló a helyzet az SMS-sel is: manapság valószínű, hogy bárkinek a világon, akinek ismered a telefonszámát, gond nélkül tudsz SMS-t küldeni, pedig lehet, hogy más a telefon gyártója, az operációs rendszere, a szolgáltatója - mégis működik.

Azonban az azonnali üzenetküldők esetén ez nem így van: Viberből nem lehet üzenetet küldeni Facebook-ra, vagy WhatsAppról Signal-ra.

Amikor kb. 23 évvel ezelőtt elkezdtem használni az ICQ-t (az akkori 98 verziót) egy darabig minden ismerősöm ezt használta, és semmi gond nem volt. Nem sokkal később azonban a Windows-ba beépülve megjelent az MSN Messenger, amit hirtelen nagyon sokan elkezdtek használni azok, akik az ICQ-val már nem találkoztak - nekik csak MSN Messengerrel lehetett üzenetet küldni, ICQ-val nem. Később ez igaz lett a Skype-ra, a Google Talk-ra, Facebook-ra, mindenre. Az okostelefonok terjedésével ez még rosszabbá vált: mára rengeteg olyan program akad, ami ráadásul kizárólag mobilon elérhető, vagy ha mást nem, legalább egy mobiltelefon-számot kér a regisztrációhoz: Whatsapp, Viber, Telegram, Signal, QQ, WeChat, stb.

A kétezres évek elején megjelentek néhány multi-protokoll program, amik megpróbálták ezt áthidalni azzal, hogy egy felületből sok rendszerhez próbáltak meg csatlakozni. Ilyen volt például a Trillian1, a Pidgin2, vagy a Miranda3. Azóta a Trillian megalkotta a saját protokollját (et tu, Brute?!), a Mirandát pedig eléggé elhanyagolták. Egyedül a Pidgin-t (illetve helyesebben szólva a programkönyvtárat mögötte, a libpurple-t) lehetséges felturbózni úgy, hogy mai, "modern" rendszereket, mint például Discord, Slack, Signal, Whatsapp4, stb. is képes legyen kezelni.

Ugyanezen évtized végén volt egy rövid időszak, amikor mind a Facebook, mind a Google, mind a WhatsApp ugyanazt a szabványt használta, amelyet XMPP5-nek hívnak. Az XMPP nem egy alkalmazás, hanem egy szabvány/protokoll - olyan, mint a HTTP és HTML - és ahhoz hasonlóan lehetséges több, másféle, nem kizárólag a hivatalos, gyártótól származó programmal használni azt. Ennek a szabványka egy elemi része kellet volna, hogy legyen az úgynevezett föderáció (federation), ami ugyanaz, mint az email esetében: mások által üzemeltetett rendszerek egymás között is tudjanak kommunikálni. Sajnos mindhárom említett szolgáltató, (Javítás: a Google Talk 2006 körül csendben engedélyezte6 egészen létezése végéig) úgy döntött, hogy ezt a funkciót kiiktatja, és sajnos azóta mindhárom szolgáltató lecserélte az XMPP-t zárt, egymással egyáltalán nem kompatibilis rendszerekre.

Közel húsz évvel ezelőtt a Firefox elérte, hogy az emberek tisztában legyenek vele, hogy az internetet nem kizárólag a két Internet Explorer programmal lehet használni, hanem bármilyen más böngészővel is: Edge, Internet Explorer, Safari, Chrome, Firefox, csak pár példát említve. Ennél sokkal kevesebben tudják, hogy ugyanez igaz (szinte) mindegyik email szolgáltatóra, beleértve a Gmail-t, az Outlookot: vannak email alkalmazások, amelyek függetlenek a szolgáltatótól, mint például Apple Mail7, K98, mutt9, Thunderbird10, Geary11 Ezzel teljes ellentétben a legtöbb azonnali üzenetküldő egyenesen tiltja azon programok használatát, ami nem az ő hivatalos programjuk - van, amelyik ezt olyan mértékben teszi, hogy visszavonhatatlanul kitiltja a felhasználót, ha ilyenen kapja.

Vannak rendszerek, amik képesek egymás között kommunikálni?

Jelenleg két fő azonnal üzenetküldő szabvány/rendszer létezik, amelyek képesek a már említett föderációra: XMPP és a Matrix.

A két rendszer között alapvető elvi különbségek vannak, ezek röviden összefoglalva:

A két megközelítés közül én az XMPP-t részesítem előnyben, de elismerem, hogy vannak előnyei a Matrix megközelítésének. Megjegyzés: a Matrix egyik funkciója, a föderált szobák, még nem létezik XMPP-n.

Hogyan lehet előrelépni a széttöredezett üzenetküldő rendszerek világából?

A válasz viszonylag egyszerű, de a kivitelezés egyáltalán nem az; ráadásul mindenkitől energiabefektetést igényel.

  1. Újra meg kell tanulnunk kisebb szolgáltatókat használni. Keress egy ismerőst, egy céget, valakit, akit személyesen ismerhetsz, valakiket, akik felveszik a telefont, ha baj van a szolgáltatással. Rengetek történet kering emberekről, akiket a Facebook kizárt az oldalról, anélkül, hogy bármilyen módon újra hozzáférhetnének a fiókjukhoz12.

  2. Válassz olyan szolgáltatót és/vagy rendszert, ami lehetővé teszi a föderációt. A https://list.jabber.at/ címen található egy lista publikus XMPP szerverekről.

  3. Válassz kliens alkalmazást szabadon, azt, ami a legjobban tetszik, nem azt, amit a szolgáltató kénye-kedve szerint, állandóan változtat. A https://xmpp.org/getting-started/ oldalon van egy lista ajánlott kliensekről.

  4. Ha képes vagy rá, futtasd a saját szolgáltatásod magadnak, barátoknak, családnak.

Modern XMPP kliensek

Az évek során - az XMPP elődje, a jabber, 1999-ben jelent meg - rengeteg új fejlesztés került az XMPP-be, de a természetéből fakadóan nem minden kliens illetve szerver támogatja az összes lehetséges funkciót. Emiatt elég sok támadás érte, holott a frissebb megoldások minden mai követelménynek megfelelnek.

Az egyik ilyen, sokat kritizált probléma, hogy sokak szerint a végpontok közötti titkosítás (e2e encryption) "túl későn" került az XMPP-be, pedig ez nem igaz: az első, PGP alapú már sok éve jelen van, de nehéz és kényelmetlen volt használni. Azóta az elsődleges megoldássá az OMEMO vált13, amit sokkal egyszerűbb használni.

A hang- és videóbeszélgetés sokáig váratott magára, de 2020-ra végre egy nyílt forráskódú Android kliens is tartalmazza a Jingle14 megoldást - megjegyzendő, hogy az AstraChat15 nevű szoftver, ami több platformra is elérhető, régóta képes erre, cserébe viszont az OMEMO titkosítást nem támogatja.

2021-re minden platformra elérhető modern, titkosítást kezelni képes, videó- és hanghívást támogató kliens.

Példák modern kliensekre

Példa: Blabber.im telepítése Androidra

Keresd ki a Play áruházból a blabber.im klienst, vagy kövesd a https://blabber.im/en/download/ linket!

blabber.im a Play Áruházban
blabber.im a Play Áruházban

Ha megvolt a telepítés, indítsd el

Bevezető

Az első néhány képernyő végigvezet az alapokon angolul. A nyilak segítségével léptesd végig. A végén használd a pipát.

img
img
img
img
img
img
img
img
img
img
img
img
img
img

Engedélyezd a file-okhoz való hozzáférést, amikor kéri!

img
img

Hozzáférés

Ezután a következő 3 képernyő elmagyarázza, hogyan működik egy hozzáférés az XMPP rendszerben.

img
img
img
img
img
img

Hozzáférés-készítés

Amikor megjelenik a "Create new account" ablak, válassz egy felhasználónevet (bármi, de angol karakterek legyenek) majd:

img
img

Ezután felajánl egy jelszót. Ezt ha akarod, elfogadhatod, de akkor írd le.

Utána ez a képernyő fogad:

img
img

Megkéri, hogy írd be a neved:

img
img

Beállítások

A következő képernyő rákérdez pár beállításra; az alapbeállítások jók, fogadd el.

img
img

Ha akarod, a következő képernyőn beállíthatsz egy avatárt.

img
img

Az ezután következő képernyő végigvezet azon, hogy hogyan tudsz beszélgetést indítani.

img
img
img
img
img
img

És megkérdezi, hogy hozzáférhet-e a névjegyeidhez.

img
img

Ha minden jól ment, kész vagy a felhasználói fiókkal.

Csatlakozás csoportbeszélgetéshez

Nyomd meg jobbra lent a nagy sárga + gombot, feljön egy menü:

img
img

Válaszd ki a Join public channel gombot:

img
img

Az felugró ablakba írd be a csoport teljes címét, pl.: chatszoba@muc.example.net

img
img
img
img
img
img

Kész vagy!

Saját XMPP szerver futtatása

Megjegyzések és figyelmeztetések

Ez nem egy kimásolható-beilleszthető útmutató; kérlek módosítsd a beállításokat. Ha elakadsz, nyugodtan vedd fel velem a kapcsolatot a mail@petermolnar.net email és/vagy XMPP címen, szívesen segítek, ha tudok.

A leírás FreeBSD FreeBSD16 szerverre készült, így a legtöbb beállítás a /usr/local/etc könyvtárra mutat a linuxban megszokott /etc helyett.

Előkövetelmény: szabadon módosítható DNS rekord beállítások

Az XMPP rendszer követelményei között szerepel több DNS rekord beállítása, így mindenképpen olyan DNS szolgáltató szükséges, amely támogatja az A, CNAME, SRV és TXT rekordokat. Az alábbi példák az example.net domaint használják; értelemszerűen módosítsd a saját domainedre.

A (és AAAA rekordok, ha ipv6 címe is van a rendszerner)

example.net. 1800 IN A a.szervered.ipv4.cime
example.net. 1800 IN AAAA a.szervered.ipv6.cime

CNAMEs a "komponens" aldomaineknek

proxy.example.net. 1800 IN CNAME example.net.
upload.example.net. 1800 IN CNAME example.net.
groupchat.example.net. 1800 IN CNAME example.net.
pubsub.example.net. 1800 IN CNAME example.net.

SRV rekordok

_xmpp-client._tcp.example.net. 1800 IN SRV 1 1 5222 example.net.
_xmpps-client._tcp.example.net. 1800 IN SRV 1 1 5223 example.net.
_xmpp-server._tcp.example.net. 1800 IN SRV 1 1 5269 example.net.

TXT rekordok

_xmppconnect.example.net. 1800 IN TXT "_xmpp-client-xbosh=https://example.net/http-bind"

Előkövetelmény: létező nginx szerver és SSL tanusítványok

Az alábbi két kategóriát ez az útmutató nem tartalmazza:

Rendkívül sok mód áll rendelkezésre mindkettőre, és kiváló útmutatók találhatóak az interneten mindkét témában.

nginx mint reverse-proxy a Prosody XMPP szerver előtt

Ezzel a lépéssel lehetővé tesszük, hogy egy pl. egy weboldalt is ki lehessen szolgálni ugyanarról a domainről, mint az XMPP szolgáltatást; vagy egy olyan webfelületet, ami az XMPP szerverünkhöz biztosít hozzáférést, mint például a movim17.

Az alábbi blokk az nginx server { } blokkján belülre kell, hogy kerüljön, a location / beállítás elé:

    # BOSH
    location /http-bind {
         proxy_pass  http://127.0.0.1:5280/http-bind;
         proxy_set_header Host $host;
         proxy_set_header X-Forwarded-For $remote_addr;
         proxy_buffering off;
         tcp_nodelay on;
    }

    # websockets
    location /xmpp-websocket {
        proxy_pass http://127.0.0.1:5280/xmpp-websocket;
        proxy_http_version 1.1;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_read_timeout 900s;
    }

    # http_upload
    location /upload {
         proxy_pass  http://127.0.0.1:5280/upload;
         proxy_set_header Host $host;
         proxy_set_header X-Forwarded-For $remote_addr;
         proxy_buffering off;
         tcp_nodelay on;
    }

Automatikusan frissülő letsencrypt tanúsítványok a Prosody részére

A certbot által szolgáltatott beállítások bővíthetőek, így lehetőségünk adódik létrehozni a /usr/local/etc/letsencrypt/renewal-hooks/post/10-prosody.sh file-t, hogy amint frissül a tanúsítvány, a Prosody szerver is automatikusan megkapja azt.

Csak akkor működik, ha a Prosody szerver tartalmaz VirtualHost elemeket.

#!/usr/bin/env bash

for d in $(cat /usr/local/etc/prosody/prosody.cfg.lua | grep VirtualHost | sed -r 's/.*"([^"]+)"/\1/'); do
        prosodyctl --root cert import "${d/.conf/}" /usr/local/etc/letsencrypt/live/
done
/usr/sbin/service prosody restart

Szerezzük be a naprakész prosody-modules csomagot

A disztribúciókban elérhető verzió általában elavult; a letöltéshez szükség van a mercurial ( hg ) verziókezelőre.

mkdir -p /opt
cd /opt
hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

Telepítsük a Prosody XMPP szervert

FreeBSD alatt:

pkg install prosody

(ha szükséges) luadbi SQLite támogatással

A FreeBSD pkg-n keresztül elérhető luadbi csomagja kizárólag a MySQL-t (abból is az 5.7-es verziót) támogatja, így ha szeretnénk SQLite adatbázist használni, le kell fordítani.

cd/usr/ports/databases/luadbi
make config
luadbi SQLite3
támogatással
luadbi SQLite3 támogatással
make
make install

A Prosody szerver beállítása

Ezeket érdemes futtatni, csak a biztonság kedvéért:

# az összes alábbi parancs root felhasználóként történik
mkdir -p /var/run/prosody
chown prosody:prosody /var/run/prosody
mkdir -p /var/db/prosody
chown prosody:prosody /var/db/prosody

# a http_upload modul részére
mkdir -p /usr/local/www/prosody
chown prosody:prosody /usr/local/www/prosody

prosody.lua.cf

plugin_paths = { "/opt/prosody-modules" }
admins = {  "admin@example.net" }

modules_enabled = {
    "announce";
    "blocklist";
    "bookmarks";
    "bosh";
    "carbons";
    "cloud_notify";
    "csi";
    "csi_simple";
    "csi_battery_saver";
    "dialback";
    "disco";
    "http";
    "http_altconnect";
    "http_files";
    "limits";
    "log_auth";
    "mam";
    "motd";
    "offline";
    "pep";
    "ping";
    "posix";
    "presence";
    "private";
    "proxy65";
    "pubsub";
    "register";
    "roster";
    "saslauth";
    "server_contact_info";
    "smacks";
    "throttle_presence";
    "time";
    "tls";
    "turncredentials";
    "uptime";
    "vcard4";
    "vcard_legacy";
    "version";
    "watchregistrations";
    "websocket";
    "welcome";
}

modules_disabled = {
}

allow_registration = false;
daemonize = true;
pidfile = "/var/run/prosody/prosody.pid";

legacy_ssl_ports = { 5223 }
legacy_ssl_ssl = {
    key = "certs/https.key";
    certificate = "certs/https.crt";
}

c2s_require_encryption = true;
s2s_require_encryption = true;
s2s_secure_auth = true;

auth_append_host = true;

pep_max_items = 10000

-- több info: https://groups.google.com/g/prosody-users/c/U1LN78jhh_A
network_default_read_size = "*a"

trusted_proxies = { "127.0.0.1", "::1" }

storage = "internal"
-- amennyiben SQLite adatbázist használnál inkább:
--sql = {
--    driver = "SQLite3";
--    database = "/var/db/prosody/prosody.db";
--}

-- naplózás
log = {
    info = "*syslog";
}

-- a tanusítványok mappája a prosody beállításokon belül
certificates = "certs";

-- BOSH (XMPP HTTP-n keresztül)
http_ports = { 5280 }
http_interfaces = { "*" }
https_ports = { 5281 }
https_interfaces = { "*" }
cross_domain_bosh = true
consider_bosh_secure = true
cross_domain_websocket = true

-- mobil kliens optimalizációk
push_notification_with_body = true
push_notification_with_sender = true
smacks_hibernation_time = 86400

-- szerver kapcsolati adatok
contact_info = {
    abuse = { "mailto:admin@example.net", "xmpp:admin@example.net" };
    admin = { "mailto:admin@example.net", "xmpp:admin@example.net" };
    security = { "mailto:admin@example.net", "xmpp:admin@example.net" };
    support = { "mailto:admin@example.net", "xmpp:admin@example.net" };
};

limits = {
    c2s = {
        rate = "100kb/s";
        burst = "2s";
    };
    s2sin = {
        rate = "100kb/s";
        burst = "5s";
    };
}

-- erre az XMPP címre riasszon a rendszer, amint új felhasználó
-- regisztrál
registration_watchers = { "admin@example.net" }

-- Proxy
-- ez egy régi kiegészítő eszköz, nagy file-ok küldésénél hasznos
-- lehet
proxy65_ports = { 5000 }
proxy65_address = "YOUR.EXTERNAL.IP.ADDRESS"

-- http_upload
-- az előző modern verziója, amikor a kliens inkább feltölti a file-t a
-- szerverre
http_upload_path = "/usr/local/www/prosody"
http_upload_file_size_limit = 10485760

-- TURN/STUN
-- audio/video hívásokra; ehhez szükséges egy külső program, lásd később
turncredentials_secret = "HOSSZÚ-TITKOS-JELSZÓ-AMIT-KÉSŐBB-MÁSHOL-IS-HASZNÁLNI-FOGUNK";
turncredentials_ttl = 86400;
turncredentials_host = "A.SZERVERED.TELJES.DOMAIN.NEVE";
turncredentials_port = 3478

-- engedélyezd ezt a beállítást, ha szeretnéd, hogy regisztrálni lehessen
-- a szerveren
-- allow_registration = true;

-- beállítás domainenként
VirtualHost "example.net"
    authentication = "internal_hashed"
    -- itt add meg hogy milyen "külső" címen érhető el a szerver; ez az, ami
    -- az nginx-ben be van állítva
    http_external_url = "https://example.net/"
    http_host = "example.net"

    -- a http_upload module részére
    Component "upload.example.net" "http_upload"
        -- ugyaúgy meg kell adni, mint az előbb; a legtöbb beállítást minden Component-en meg
        --- kell adni külön
        http_external_url = "https://example.net/"
        http_host = "example.net"

    -- a proxy részére
    Component "proxy.example.net" "proxy65"

    -- csoportbeszélgetések részére - enélkül nincs lehetőség szobákat nyitni
    Component "groupchat.example.net" "muc"
        modules_enabled = {
            "muc_mam",
            "vcard_muc"
        }
   -- amennyiben szükséges, pl. movim - https://movim.eu/
   -- részére
   -- Component "pubsub.example.net" "pubsub"
   --    pubsub_max_items = 10000

Futtassuk az alábbi parancsokat, hogy betöltsük a tanusítványokat:

# töltsük be a meglevő tanusítványokat az első akalommal az
# előkészített letsencrypt kiegészítő parancsból:
bash /usr/local/etc/letsencrypt/renewal-hooks/post/10-prosody.sh

# készítsünk symlinket a fő tanusítványra, hogy az ún. legacy SSL modulok
# tudjanak mit használni
cd /usr/local/etc/prosody/certs
ln -s example.net.crt https.crt
ln -s example.net.key https.key

Indítst a Prosody szervert

service prosody start

Ha nem engedélyezted a regisztrációt, a prosodyctl paranccsal lehet felhasználót létrehozni:

prosodyctl adduser admin@example.net

(optionális) coturn TURN szerver a telepítése

Videó- és hanghívások támogatásához.

pkg install coturn

(optionális) coturn TURN szerver beállítása

Az alábbi beállítások egy olyan szerverre érvényesek, amelyik egyetlen, közvetlenül az internetre kapcsolt IP címmel rendelkeznek. Amennyiben ez nem igaz, a beállításokat ehhez mérten kell módosítani.

/usr/local/etc/turnserver.conf

listening-port=3478
tls-listening-port=5349
listening-ip=0.0.0.0
min-port=49152
max-port=65535
use-auth-secret
static-auth-secret=HOSSZÚ-TITKOS-JELSZÓ-AMIT-KÉSŐBB-MÁSHOL-IS-HASZNÁLNI-FOGUNK
server-name=example.net
realm=example.net
cert=/usr/local/etc/prosody/example.net.crt
pkey=/usr/local/etc/prosody/example.net.key
syslog
proc-user=prosody
proc-group=prosody

Tűzfal beállítások

Az alábbi portok mindeképpen szükségesek, hogy az XMPP szerver helyesen üzemeljen:

ipfw add 03600 allow tcp from any to me 5222
ipfw add 03700 allow tcp from any to me 5223
ipfw add 03800 allow tcp from any to me 5269
ipfw add 04100 allow tcp from any to me 5000

Amennyiben a TURN szerver is beállításra került:

ipfw add 04700 allow tcp from any to me 3478
ipfw add 04800 allow udp from any to me 3478
ipfw add 04900 allow tcp from any to me 5349
ipfw add 05000 allow udp from any to me 5349
ipfw add 05200 allow udp from any to me 49152-65535
ipfw add 05300 allow tcp from any to me 49152-65535

Az /etc/rc.conf file beállításában, hogy újraindítás után is beállításra kerüljenek:

firewall_script="/etc/rc.firewall"
firewall_type="workstation"
firewall_myservices="22/tcp 5222/tcp 5223/tcp 5269/tcp 5000/tcp 3478/tcp 3478/udp 5349/tcp 5349/udp 49152-65535/udp 49152-65535/tcp"
firewall_allowservices="any"
firewall_logging="YES"
firewall_logdeny="YES"
firewall_enable="YES"
firewall_quiet="YES"

Megjegyzés: a 22-es port a biztonság kedvéért van a listában, nehogy valaki kizárja magát a rendszerből.

Javaslat: érdemes a rc.conf file firewall_ részéről bővebben olvasni, mielőtt az ember alkalmazná, mert könnyű kizárni magunkat a rendszerből.

Végszó

Amíg használni az XMPP könnyű, egy szervert üzemeltetni egyáltalán nem az. Szerencsére a Prosody XMPP szerver rendszerigénye rendkívül alacsony, és a mai, gyors internetkapcsolatokkal akár otthon is, egy régi laptopon, vagy egy Raspberry Pi-n is lehetséges egy teljes értékű szervert üzemeltetni. Ha nem áll rendelkezésre fix IP, az sem baj: a DigitalOcean18 például ingyenes DNS szervert biztosít, amin egyszerű HTTP hívásokkal keresztül lehet módosítani a rekordokat, amennyiben frissül az IP cím.

A https://prosody.im/doc rengeteg, hasznos dokumentációkat tartalmaz, de sajnos kissé nehéz navigálni az oldalon.

Ha le akarod tesztelni a szervered, a https://compliance.conversations.im/ segít benne.

Amennyiben hibát találsz a leírásban, kérlek jelezd, de szívesen fogadok bővítést, kiegészítést is.

Sok sikert!


  1. https://www.trillian.im/

  2. https://pidgin.im/

  3. https://sourceforge.net/projects/miranda/

  4. https://github.com/petermolnar/awesome-pidgin-plugins

  5. https://xmpp.org/

  6. http://googletalk.blogspot.com/2006/01/xmpp-federation.html

  7. https://www.lifewire.com/set-up-gmail-account-with-macs-mail-application-2260069

  8. https://k9mail.app/documentation/accounts/providerSettings.html

  9. http://www.mutt.org/

  10. https://www.thunderbird.net/

  11. https://wiki.gnome.org/Apps/Geary

  12. https://www.elliott.org/blog/banned-from-facebook-permanently-how/

  13. https://omemo.top/

  14. https://prosody.im/doc/jingle

  15. https://astrachat.com/

  16. https://www.freebsd.org/

  17. https://movim.eu/

  18. https://www.digitalocean.com/

by Peter Molnar <mail@petermolnar.net>

DIY smart wake up lamp with IKEA, Sonoff D1 Dimmer, and Domoticz

I made a smart wake up lamp, just to see if it is better, than a normal alarm clock. It is an insane difference, waking up is so much nicer, and subtle, than before, so I recommend doing one.

WARNING: this tutorial contains works on a unit that runs on mains electricity. Be extremely aware that mains electricity is dangerous, and could kill you, or anyone exposed to it.

Please also note that in certain countries, like Australia, you need to be a licensed electrician to deal with mains electricity, or you could be heavily fined.

NEVER, EVER TAKE ANYTHING APART THAT IS STILL CONNECTED TO MAINS.

DO NOT CONNECT THE DIMMER TO THE MAIN ELECTRICITY UNTIL IT'S SAFE

I am not liable for any damage or injury that could happen due to following this tutorial


Why not a smart bulb?

When I started out with Domoticz I wanted to get into Z-Wave, because from a tech perspective, it's far superior to the rest - or so I thought. However, the only z-wave equipment I ended up with was our thermostat and boiler controller, and these two turned out to be a lousy choice:

The reason for not making everything into z-wave was due to the bad experiences with these ones, and because it's way more expensive, than anything else.

Before getting into this project, I bought an IKEA smart bulb, namely a TRÅDFRI white spectrum dimmable one. I'm not going to link it, because it's not a good product (especially for the purpose of a wake up lamp), for two reasons:

The other problem was that it's Zigbee. There is no common zigbee protocol over USB, and it's an utter mess to get zigbee properly working with domoticz: one needs a good enough zigbee sniffer, an open source firmware on it, an mqtt server, and a node.js service that translates between mqtt and zigbee. In comparison: rflink, a free arduino mega firmware for 433MHz bridging is essentially plug-and-play.

I could have gone for flashing ESPHome1 or Tasmota2 on a light bulb that might be running on Tuya, but I got tired of "maybe I get lucky and receive the device I actually ordered on eBay".

And this is how I arrived at the Sonoff D1: it's a real dimmer, it can accept any dimmable bulb, and it runs on WiFi.

Sonoff D1 Dimmer3

I've been eyeing the Sonoff products for a while, but until I had an actual use case for them, I wasn't going to buy one. With the need for a wake up lamp, I bumped into the D1, and it is a very decent device - except for one thing, see later.

DIY mode4

Newer Sonoff devices have something called a "DIY" mode: this turns the sonoff device's web interface to accept API calls directly. It's a simple, reversible, official process, and you'll end up with a device that can be easily commaneded with mere HTTP calls.

Note: to turn on DIY mode on the D1 you still the 433MHz remote controller.

Troubles with the 433MHz remote

The only trouble with D1 is with it's remote controller. The remove is a 433MHz, good looking thing, that can even be mounted as a light switch on the wall. There's but one funny thing: apparently it sometimes turns on on it's own5. This can easily result in the lamp coming on full 100% brightness at 4am. Yes, it did happen to me.

One solution is crude, but works: un-pair the remote from the D1. After this, if you have a 433MHz bridge, like and rflink6 you could map the buttons to devices in domoticz and trigger and API call to the D1, so the remote still functions as intended. Since un-pairing the remote, I had no fun surprises in the middle of the night.

There might be another solution which involves adding an rc filter to the device7 but I didn't want to solder just yet, so I can't verify if it works.

The base: an old(ish) IKEA lamp

I have two IKEA SÅNGEN Table lamps that have been out of production for a while. The good thing about them is that they have an open bottom base with plenty of space in there to attach the D1.

IKEA SÅNGEN Table lamp
license: IKEA SÅNGEN Table lamp
Camera
PENTAX K-5 II s
Aperture
f/4.5
Shutter speed
1/25 sec
Focal length (as set)
60.0 mm
Sensitivity
ISO 1600
Lens
HD PENTAX-DA 16-85mm F3.5-5.6 ED DC WR
Sonoff D1 attached to the base of an IKEA SÅNGEN with double sided
tape.
license: Sonoff D1 attached to the base of an IKEA SÅNGEN with double sided tape.
Camera
PENTAX K-5 II s
Aperture
f/4.5
Shutter speed
1/30 sec
Focal length (as set)
48.0 mm
Sensitivity
ISO 800
Lens
HD PENTAX-DA 16-85mm F3.5-5.6 ED DC WR

Notice the tape over the connectors if the D1: you MUST cover the electrical connectors with insulating tape! This is no joke. The D1 was never planned to be use like this, it was made to be an internal or in wall element. Without the tape it exposes barely sunk mains cables:

If you leave it like this it could kill
someone.
license: If you leave it like this it could kill someone.
Camera
PENTAX K-5 II s
Aperture
f/4.5
Shutter speed
1/30 sec
Focal length (as set)
43.0 mm
Sensitivity
ISO 800
Lens
HD PENTAX-DA 16-85mm F3.5-5.6 ED DC WR
Use the tape generously. Cover every inch that could expose the live
wires.
license: Use the tape generously. Cover every inch that could expose the live wires.
Camera
PENTAX K-5 II s
Aperture
f/4.5
Shutter speed
1/30 sec
Focal length (as set)
60.0 mm
Sensitivity
ISO 1600
Lens
HD PENTAX-DA 16-85mm F3.5-5.6 ED DC WR
Position it in a corner, so it becomes even harder for anyone to come
in contact with the wires.
license: Position it in a corner, so it becomes even harder for anyone to come in contact with the wires.
Camera
PENTAX K-5 II s
Aperture
f/4.5
Shutter speed
1/30 sec
Focal length (as set)
60.0 mm
Sensitivity
ISO 1600
Lens
HD PENTAX-DA 16-85mm F3.5-5.6 ED DC WR

I also replaced the bulb fixture: the original it was an E14, and most of the bulbs in our home are B22; it's simpler to have everything as B22.

The replaced bulb fixture.
license: The replaced bulb fixture.
Camera
PENTAX K-5 II s
Aperture
f/5.6
Shutter speed
1/30 sec
Focal length (as set)
85.0 mm
Sensitivity
ISO 1600
Lens
HD PENTAX-DA 16-85mm F3.5-5.6 ED DC WR

I opted for a halogen bulb: unlike LED, halogen can be properly dimmed, from near no light to full brightness.

An old halogen bulb with B22
bayonette
license: An old halogen bulb with B22 bayonette
Camera
PENTAX K-5 II s
Aperture
f/4.5
Shutter speed
1/30 sec
Focal length (as set)
60.0 mm
Sensitivity
ISO 1600
Lens
HD PENTAX-DA 16-85mm F3.5-5.6 ED DC WR

Domoticz

Plugin for the Sonoff DIY mode

Not so surprisingly nobody fiddled with the D1 Dimmer in DIY mode wired into Domoticz, so I wrote a plugin - consider the plugin to be in beta stage, given I've been running it for a few months now, without any real issues (well, once the initial bugs were ironed out). It's also on Github8.

This is a Python plugin; your Domoticz has to support Python plugins in order to work.

/path/to/your/domoticz/dir/plugins/sonoff-d1-diy/plugin.py

__author__ = "Peter Molnar"
__maintainer__ = "https://petermolnar.net"
__email__ = "mail@petermolnar.net"

"""
<plugin key="sonoff_d1_diy" name="Sonoff D1 Dimmer DIY connector" version="0.1">
    <description>
        <h2>Sonoff D1 Dimmer DIY connector</h2><br/>
    </description>
    <params>
        <param field="Address" label="Local IP Address" width="200px" required="true" default="192.168.0.1"/>
        <param field="Mode1" label="Port" width="200px" default="8081"/>
        <param field="Mode6" label="Debug" width="150px">
            <options>
                <option label="None" value="0" default="true" />
                <option label="Python Only" value="2"/>
                <option label="Basic Debugging" value="62"/>
                <option label="Basic+Messages" value="126"/>
                <option label="Connections Only" value="16"/>
                <option label="Connections+Python" value="18"/>
                <option label="Connections+Queue" value="144"/>
                <option label="All" value="-1"/>
            </options>
        </param>
    </params>
</plugin>
"""
import Domoticz
import json


class BasePlugin:
    httpConn = None
    oustandingPings = 0
    connectRetry = 3

    def __init__(self):
        return

    def onStart(self):
        if Parameters["Mode6"] != "0":
            Domoticz.Debugging(int(Parameters["Mode6"]))

        self.httpConn = Domoticz.Connection(
            Name=Parameters["Address"],
            Transport="TCP/IP",
            Protocol="HTTP",
            Address=Parameters["Address"],
            Port=Parameters["Mode1"],
        )
        self.httpConn.Connect()

    def onStop(self):
        self.httpConn.Disconnect()
        self.httpConn = None
        Domoticz.Log("onStop - Plugin is stopping.")

    def onConnect(self, Connection, Status, Description):
        if Status == 0:
            Domoticz.Debug("Connected to Sonoff DIY interface")
            self.query_status(Connection)
        else:
            Domoticz.Log(
                "Failed to connect ("
                + str(Status)
                + ") to: "
                + Parameters["Address"]
                + ":"
                + Parameters["Mode1"]
                + " with error: "
                + Description
            )

    def onMessage(self, Connection, Data):
        try:
            strData = Data["Data"].decode("utf-8", "ignore")
            strData = json.loads(strData)
            # Status = int(Data["Status"])
            Domoticz.Debug("Parsed JSON:" + json.dumps(strData))
            self.oustandingPings = self.oustandingPings - 1
        except:
            Domoticz.Error("Failed to parse response as JSON" + strData)
            return

        if "data" in strData and "deviceid" in strData["data"]:
            return self.update_device(strData["data"])


    def onCommand(self, Unit, Command, Level, Hue):
        Domoticz.Debug(
            "onCommand called for Unit "
            + str(Unit)
            + ": Parameter '"
            + str(Command)
            + "', Level: "
            + str(Level)
        )

        # in case of 'on' and 'off', the dimmable endpoint doesn't seem to respect the switch status
        if "on" == Command.lower():
            url = "/zeroconf/switch"
            data = {"switch": "on"}
            n_value = 1
        elif "off" == Command.lower():
            url = "/zeroconf/switch"
            data = {"switch": "off"}
            n_value = 0
        else:
            url = "/zeroconf/dimmable"
            if Level > 0:
                switch = "on"
                n_value = 1
            else:
                switch = "off"
                n_value = 0
            data = {"switch": switch, "brightness": Level}
        self.httpConn.Send(
            {
                "Verb": "POST",
                "URL": url,
                "Headers": {"Content-Type": "application/json"},
                "Data": json.dumps({"deviceid": "", "data": data}),
            }
        )
        self.query_status(self.httpConn)

    def onDisconnect(self, Connection):
        Domoticz.Log(
            "onDisconnect called for connection to: "
            + Connection.Address
            + ":"
            + Connection.Port
        )

    def onHeartbeat(self):
        try:
            if self.httpConn and self.httpConn.Connected():
                self.oustandingPings = self.oustandingPings + 1
                if self.oustandingPings > 6:
                    Domoticz.Log(
                        "Too many outstanding connection issues forcing disconnect."
                    )
                    self.httpConn.Disconnect()
                    self.nextConnect = 0
                else:
                    self.query_status(self.httpConn)
            elif self.httpConn:
                # if not connected try and reconnected every 3 heartbeats
                self.oustandingPings = 0
                self.nextConnect = self.nextConnect - 1
                if self.nextConnect <= 0:
                    self.nextConnect = 3
                    self.httpConn.Connect()
            else:
                self.onStart()
            return True
        except:
            Domoticz.Log(
                "Unhandled exception in onHeartbeat; resetting"
            )
            self.httpConn = None
            self.onStart()

    def update_device(self, data):
        # create new devices if the don't exist just yet
        existing_devices = [d.DeviceID for d in Devices.values()]
        if data["deviceid"] not in existing_devices:
            # I guess brightness is only present in a dimmer
            # I could be wrong
            if "brightness" in data:
                Domoticz.Device(
                    Name=data["deviceid"],
                    Unit=1,
                    Type=244,
                    Subtype=73,
                    Switchtype=7,
                    DeviceID=data["deviceid"],
                ).Create()

        # now the device certainly exists, so find it
        device = None
        for index, d in Devices.items():
            if data["deviceid"] == d.DeviceID:
                device = Devices[index]

        if not device:
            Domoticz.Error("something is wrong: the device was not found?!")
            return

        if "switch" in data and "brightness" in data:
            if data["switch"] == "on":
                n_value = 1
            else:
                n_value = 0
            s_value = str(data["brightness"])
            # SignalLevel: see https://stackoverflow.com/a/31852591
            device.Update(
                nValue=n_value,
                sValue=s_value,
                SignalLevel=min(
                    max(2 * (data["signalStrength"] + 100), 0), 100
                ),
                BatteryLevel=100,
            )

    def query_status(self, Connection):
        Connection.Send(
            {
                "Verb": "POST",
                "URL": "/zeroconf/info",
                "Headers": {"Content-Type": "application/json"},
                "Data": json.dumps({"data": ""}),
            }
        )

global _plugin
_plugin = BasePlugin()


def onStart():
    global _plugin
    _plugin.onStart()


def onStop():
    global _plugin
    _plugin.onStop()


def onConnect(Connection, Status, Description):
    global _plugin
    _plugin.onConnect(Connection, Status, Description)


def onMessage(Connection, Data):
    global _plugin
    _plugin.onMessage(Connection, Data)


def onCommand(Unit, Command, Level, Hue):
    global _plugin
    _plugin.onCommand(Unit, Command, Level, Hue)


def onNotification(Name, Subject, Text, Status, Priority, Sound, ImageFile):
    global _plugin
    _plugin.onNotification(
        Name, Subject, Text, Status, Priority, Sound, ImageFile
    )

def onDisconnect(Connection):
    global _plugin
    _plugin.onDisconnect(Connection)


def onHeartbeat():
    global _plugin
    _plugin.onHeartbeat()
You'll need to add a new hardware device for each
D1
You'll need to add a new hardware device for each D1

DzVents wake up script

Unless I overlooked it, there isn't a simple functionality to add custom timed dimmer in domoticz, but it's simple to add one as a DzVents event script:

return {
    on = {
        timer = {
            'at 7:30 on mon,tue,wed,thu,fri'
        }
    },
    execute = function(domoticz, timer)
        domoticz.log('Timer event was triggered by ' .. timer.trigger, domoticz.LOG_INFO)
        -- replace '742' with the id of your D1
        local dimmer = domoticz.devices(742)
        local level = 0
        -- this is ugly, because it fires literally 100 events
        -- to be executed with 3 seconds from eachother for
        -- each % level of brightness - but it works
        -- feel free to alter this for a more subtle or more powerful
        -- wake up experience
        while (level < dimmer.maxDimLevel) do
            level = level+1
            dimmer.dimTo(level).afterSec(level*3)
        end
        -- these were the tricky ones to find:
        -- dim to 1 and not 0, because 0 means turn off, but sadly
        -- doesn't actually set the dim
        dimmer.dimTo(1).afterMin(44)

        -- make sure it's off once it's dimmed to 1%
        dimmer.switchOff().afterMin(45)
    end
}

Closing words

This was surprisingly fun to make. The DIY mode in Sonoff is reasonable to work with, and if it wasn't for the bug of having the light turning on randomly when paired with the remote, this would be an incredibly nice, and affordable device. This was it's a good, and affordably device.

Don't fiddle with dimmable LEDs; just get a dimmable halogen bulb. I know it eats more, but it also gives off a more natural (might even say healthier) light9, which will make it a better wake up light.


  1. https://esphome.io/

  2. https://tasmota.github.io/docs/Tuya-Convert/

  3. https://amzn.to/36PZfCQ

  4. http://developers.sonoff.tech/sonoff-diy-mode-api-protocol.html

  5. https://support.itead.cc/support/discussions/topics/11000030765

  6. http://rflink.nl/blog2/

  7. https://www.youtube.com/watch?v=aq8_os6g13s

  8. https://github.com/petermolnar/domoticz-sonoff-d1-diy

  9. https://www.researchgate.net/publication/330372622_Recent_Progress_in_Solar_Cell_Technology_for_Low-Light_Indoor_Applications

by Peter Molnar <mail@petermolnar.net>

Targeted ads narrow your world

This post is a reply to: https://news.ycombinator.com/item?id=25990825

Long, long time ago, when last.fm radio was still a thing on it's own, I loved it: instead of pre-arranging a playlist, it kept adding the next song based by searching for similar ones based on the one currently playing. It kept crawling away, sometimes into terrible direction, but more often into an area I'd like to refer to as "satellite".

"Satellite" would be friends-of-friends or even friends-of-friends-of-friends in social media terms.

And this is how it comes to ads: when I bought a magazine, I got a set of ads that only had a rough idea about their audience. Those edges were the ones that allowed ads to broaden my views on the world, and, therefore, ads were useful for me. This, and the financial impossibility of ever getting one - early teen in Hungary -, were the reasons why I loved Sony catalogues in the '90s.

Targeted ads are the polar opposite: they want to change me, force me to buy a certain, specific product, instead of gradually opening my interest to a lot more things in the world.

by Peter Molnar <mail@petermolnar.net>

Budget-friendly webcam upgrade: smartphone with DroidcamX over USB

I found a program, called Droidcam, that can turn any Android phone into a surprisingly nice webcam over USB, and it works remarkably well even on linux.

Note: nobody paid me for this article, I merely want to share my experiences, but some links at the bottom are amazon affiliate links.

DroidCamX

Today is March 333, 2020: I'm still in a full-on lockdown in the UK, and all my life is happening through online video, just like for the past nearly one year. I've put many of my activities on hold, including progressing with my martial studies1, but considering there doesn't seem to be an end to any of it soon, it's time to make it work through the dreaded video. (I'm not against videochat, but doing martial arts, even tai chi, alone, on video conferencing is anything but ideal, if one wants to learn, correct their mistakes, and progress.)

I found that my laptop's webcam quality was simply not up to the game: slow frame rate, low resolution, terrible low light performance. It's ok for daily meetings, but not for capturing movement. At first I was looking into building a Raspberry Pi Zero webcam2 with a high quality lens, but I was reluctant to spend a considerate amount of money on it (it would have been over £100), and to add yet another device to my home.

I had countless rounds trying to turn some of my former smartphones into a security camera, so I had some experience with IP camera apps, and I was never completely happy with the outcome. Recently, however, I started to see reviews around apps that turn iPhones into exceptionally good webcams - Camo3 and EpocCam4 to be specific -, using USB. The only problem is: I have android(s).

In the end I somehow found Droidcam5, and it's paid version, DroidCamX6: an app/software combo, that allows Windows and Linux (sic) users to use their devices as webcam. The paid upgrade is to allow HD and fullHD resolution; for a one-off £4.99, it's a good deal.

Their website has perfect description on how to set it up at: https://www.dev47apps.com/

The step-up from the laptop's webcam is formidable:

This camera quality of a Lenovo ThinkPad X250's built-in webcam -
"SunplusIT" according to lshw -, in natural light. This is the full,
native, 848x480 resolution, captured with
VLC
This camera quality of a Lenovo ThinkPad X250's built-in webcam - "SunplusIT" according to lshw -, in natural light. This is the full, native, 848x480 resolution, captured with VLC
A Moto E5 phone, using Droidcamx, connected via USB, set to 1920x1080
resolution; same time, same day, same spot, though the camera was \~5cm
closer to me.
A Moto E5 phone, using Droidcamx, connected via USB, set to 1920x1080 resolution; same time, same day, same spot, though the camera was ~5cm closer to me.

I've tried using it on an old android, namely a Samsung Galaxy Nexus. I had to install an old CyanogenMod version7 on it to have Android 5.1, but it worked just fine afterwards.

The last official Android version for this phone was 4.38 and the camera API was altered considerably in 5.0 - which is the minimum requirement for DroidCam. I've tried a have a custom ROM based on android 7 on the device, Droidcam only resulted in a blank, black screen, so it's safe to conclude that one will need a device that was released with Android 5.0 or higher.

Wide angle

Normally my camera angle is enough, but I'm planning to move my workouts to a much smaller space where I can hang a punching bag. I've had a round with a wide angle lens at the start of the pandemic, which I ended up sending back, given it cut off about 1/4 of the view due to being too small.

This time I spent a lot time on reading review, focusing on the wide-angle and not the macro experience - that is because nearly every clip-on lens out there is at least a 2-in-1: a macro and a wide angle.

In the end I've found a 140º Wide Angle Camera Lens on amazon9, costing £16.79 in a sale, and I was pleasantly surprised:

LUXSURE Professional wide angle lens - it's bigger, and heavier, than
it looks
LUXSURE Professional wide angle lens - it's bigger, and heavier, than it looks
A Moto E5 phone, using Droidcamx, connected via USB, set to 1920x1080
resolution; same time, same day, same spot at the other Moto E5 photo
from before, but with a wide-angle clip on
lens
A Moto E5 phone, using Droidcamx, connected via USB, set to 1920x1080 resolution; same time, same day, same spot at the other Moto E5 photo from before, but with a wide-angle clip on lens

Obviously, with similar any addon lens, it will make the image quality is a bit worse, but there's no vignetting, no dark corners. The barrel distortion is quite heavy, but I wasn't expecting miracles for that price - in comparison, a real wide-angle lens for a Pentax camera is well into the £600 range at minimum.

It's important to note that if it's not positioned correctly, the picture will get very blurry; if this happens, make sure that the center of the lens is directly above the center of camera - not the camera bump/area, but the hole, which is the camera itself.

Stand/mount

I didn't have any kind of stand/mount/holder for my phone, that could hold the position required, or be mounted on a tripod. When I started looking most results where for cheap, basically single use, plastic mounts.

I'm very tired of buying equipment that fails me way too soon: in comparison, my trusty Slik Sprint tripod is from 2006, and still going strong, despite having experienced seawater, mud, rain, and a lot of abuse.

So I decided to go for something that looks like it might last, and payed £24.99 for a metal phone holder10, instead of ~£9 plastic one. So far so good - even the tightening knob is metal!

Woohoto is not yet a well respected photo gear brand, but if they keep
producing this kind of quality, they could
be
Woohoto is not yet a well respected photo gear brand, but if they keep producing this kind of quality, they could be

Notes

There were quite a few articles recently around the topic that webcam quality is terrible, even for the expensive ones11 compared to basically any current day phone. This lead me to avoid the search for a webcam and explore what could be done with some upgrade to my existing equipment. DroidCamX works very nice.

The only downside is that encoding a 480x360 video stream is not the same as encoding 1920x1080 - your machine will certainly work harder during the video conferences. If it can't cope with it, consider lowering the resolution, because even if the resolution is the same, the image quality is still significantly better, than of a built-in laptop webcam.


  1. https://pakua.com

  2. http://www.davidhunt.ie/raspberry-pi-zero-with-pi-camera-as-usb-webcam/

  3. https://reincubate.com/camo/

  4. https://www.elgato.com/en/epoccam

  5. https://www.dev47apps.com/

  6. https://play.google.com/store/apps/details?id=com.dev47apps.droidcamx

  7. https://cloud.petermolnar.net/index.php/s/9Xdbjopr9GjDz6Z

  8. https://developers.google.com/android/images#yakju

  9. https://amzn.to/3qW8FEx

  10. https://amzn.to/2MnBZot

  11. https://reincubate.com/support/how-to/why-are-webcams-bad/

by Peter Molnar <mail@petermolnar.net>

Run your artisan instant messaging service for your friends & family

Fleeing Whatapp due to their policy changes? Tired of the never ending roam between the actual trendy instant messaging app? Here is a simple, elegant solution: run your own server, onboard your friends and family, and live happily ever after.

Chat systems should be able to talk across one another, why are they not?

You probably have used email before. I have multiple email addresses, among them mail@example.net and petermolnar.eu@gmail.com . It's perfectly normal that I will be able to send a mail from one another, even though these email addresses are on different domain (example.net vs gmail.com), different servers, and different are the people running them.

There's a similar case with SMS: nowadays I expect that I'd be able to send a text to anyone in the world who's number I know, and if I'm using the international number format correctly. +44 (UK) to +36 (Hungary), or +49 (Germany) to +1 (USA) - it should all work. I have to admit, I don't know if sending a text to mainland China works; I never tried, but I'd believe it does.

When it comes to instant messages, this was never the case, even though it has been possible for a long time - just disabled by most providers.

Around 23 years ago I launched ICQ 98, and the people I could interact with were the people on ICQ. If they were on AIM, I'd need to launch AIM. Later MSN, Skype, Gtalk came onto play, all before mobile took off.

Somewhere down the road fantastic developers had enough of needing to use many software for the same purpose, and thus multi-protocol instant messengers happened, notably: Trillian1, Pidgin2, Miranda3. Since then Trillian became yet another instant messaging protocol (et tu, Brute?!), Miranda got heavily neglected. Pidgin, or, more specifically, libpurple, the library behind it, is doing acceptably well, and can be made to connect to "modern" walled garden services, such as Discord, Slack, Signal, or even Whatsapp4

This is how my Pidgin, my multi-protocol application looked in 2020.
Wouldn't it be simpler to have only one for work, and another for
personal?
This is how my Pidgin, my multi-protocol application looked in 2020. Wouldn't it be simpler to have only one for work, and another for personal?

When smartphones started to become widely available, the messenger space grew exponentially, fuelled by "free" SMS and calls. Now we have more instant messengers, then ever, and they can only, and strictly only be used within the same system, many are mobile "exclusive": Whatsapp, Viber, Telegram, Signal, new ICQ, Google Talk Hangouts Allo Duo Meet Chat (I think it's called Chat at the moment), Threema, Facebook Messenger, QQ, WeChat - the list is endless.

There was a brief, interesting period of time, namely when Facebook introduced their chat, Gtalk was still called Gtalk, and Whatsapp was still just starting: they all used the very same underlying standard, called XMPP5. XMPP is not an app; it's more like HTTP and HTML; a wire frame, so people can build their own apps on top of it. It has a feature that sadly none of the three mentioned service provided, called federation: it means that just like email, it should have been possible to talk from your Whatsapp with someone on Gtalk. Since then, all three of them abandoned XMPP, and made different, incompatible with each other systems.

This feature then was disabled (EDIT: with the exception of Google Talk in 2006, that had federation until it's demise6); probably to train people to associate an app with the service behind it. Many of us know that the internet can be accessed with many browsers - Edge, Internet Explorer, Safari, Chrome, Firefox. Much less know that nearly ever email service, including Gmail, or Outlook, can also be accessed with different programs, like Apple Mail7, or K98 on android. With instant messengers, using a 3rd party client sometimes even leads to instant ban. If you think of it, it's completely insane.

What can be done to address the fragmented nature of instant messaging?

The answer is simple, but the execution is not at all, and it needs effort from all of us.

  1. We need to return to smaller service providers. Find a firm where you'll know real people, where you'll know the faces to turn to if you have a problem. There are so many stories about people locked out from their accounts on Facebook without any option to appeal that it's frightening9. These days places like https://conversations.im/ is a good start for such provider.
  2. Use a provider which allows your account to talk to other accounts on the same network - it's called federation. (See below)
  3. Choose an app you like (see later), not one someone wants to force on you.
  4. If you can, run your own service, and do it for friends, and family as well.

What are the networks capable of talking to each other?

At the moment there are two main messaging ecosystems which can do federation and accepts many clients: XMPP and Matrix. They cannot (yet?) talk to one another. There are fundamental philosophical differences in their approach, which can be summarised as:

I prefer the approach of XMPP.

About modern XMPP clients

One of the beautiful benefits of using a service that allows, sometimes even encourages, using the application you prefer, is that we can decide based on our needs and likes.

There have been many critical voices aimed at XMPP, mainly like "too late to add end-to-end encryption". This is a complex question, because in fact XMPP was one of the first things to offer encryped messaging - PGP based -, but it was very difficult to use. A more modern approach, borrowed from Signal, arrived later, and it's called OMEMO. There are now many client supporting OMEMO encryption10, providing basically transparent, secure chat to many.

Another feature that many waited a long time was voice and video chat - but this, again, has a similar story to encryption. Jingle11 is a very old addition to the XMPP standard, so old, that it doesn't work with Pidgin any more - the relevant Pidgin part was built around a media library that has been dead for over a decade, namely gstreamer-0.1 -, but on mobile clients, this feature was missing for a long time. A notable exception is AstraChat12 which offered it for quite a while, but in return, it still doesn't offer e2e encryption.

The point is: in 2020, XMPP clients on mobile or desktop are up to all needs, all requirements, and are certainly much easier to use, then they used to be even just a few years ago.

The clients I'd recommend these days

With a heavy heart, I'm not putting Pidgin on that list, because it's basically impossible to get audio/video support for XMPP on it, and while I'll remain a heavy user of it due to needing to connect to many networks, it is not the best choice if someone is looking for a good, modern XMPP client.

There's an ever growing list of clients at https://xmpp.org/software/clients.html for more options.

aTalk has an incredible FAQ at https://atalk.sytes.net/atalk/faq.html which probably answers any and all questions one might have with XMPP from the user side.

Running a service: prosody behind nginx

Notes and warnings

This is how I've done it; it's not guaranteed it'll work for you. If you get stuck, or want to hire me to set it up for you, reach out to me at mail@petermolnar.net (that's both an email and an XMPP address).

My server runs FreeBSD13, meaning most paths will be /usr/local/etc instead of /etc as it would be on Debian or CentOS.

Prerequisites: access to DNS configuration

Running a webserver is usually simple, but those who'd ever set up an email server know there are additional magic that needs to be added to the domain records. The case of an XMPP server is not as complex as with email, but neither is so simple, with a mere A record.

Base A (and AAAA records, if possible)

example.net. 1800 IN A your.server.ipv4.address
example.net. 1800 IN AAAA your.servers.ipv6.address

CNAMEs for subdomains used by components

proxy.example.net. 1800 IN CNAME example.net.
upload.example.net. 1800 IN CNAME example.net.
groupchat.example.net. 1800 IN CNAME example.net.
pubsub.example.net. 1800 IN CNAME example.net.

Service records for the XMPP clients

_xmpp-client._tcp.example.net. 1800 IN SRV 1 1 5222 example.net.
_xmpps-client._tcp.example.net. 1800 IN SRV 1 1 5223 example.net.
_xmpp-server._tcp.example.net. 1800 IN SRV 1 1 5269 example.net.

TXT record for BOSH

_xmppconnect.example.net. 1800 IN TXT "_xmpp-client-xbosh=https://example.net/http-bind"

Prerequisites: nginx and SSL (letsencrypt) certificates

There are two topics not covered in this mini tutorial:

There are many ways of doing both, and excellent tutorials are available.

nginx configuration to act as reverse-proxy for Prosody

Nginx is an ideal reverse proxy. By putting it in front of Prosody, we keep the ability to serve websites, or web interfaces for our XMPP server, like movim14.

This is the minimum required configuration that goes inside a server { } block, before the location / part for nginx to forward everything needed for Prosody:

    # BOSH
    location /http-bind {
         proxy_pass  http://127.0.0.1:5280/http-bind;
         proxy_set_header Host $host;
         proxy_set_header X-Forwarded-For $remote_addr;
         proxy_buffering off;
         tcp_nodelay on;
    }

    # websockets
    location /xmpp-websocket {
        proxy_pass http://127.0.0.1:5280/xmpp-websocket;
        proxy_http_version 1.1;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_read_timeout 900s;
    }

    # http_upload
    location /upload {
         proxy_pass  http://127.0.0.1:5280/upload;
         proxy_set_header Host $host;
         proxy_set_header X-Forwarded-For $remote_addr;
         proxy_buffering off;
         tcp_nodelay on;
    }

Letsencrypt post-renewal hook

I've added this into /usr/local/etc/letsencrypt/renewal-hooks/post/10-prosody.sh so whenever the certbot requests new certificates automatically, it'll get deployed for prosody.

This needs VirtualHost in Prosody to work; if you're not using that, you'll need to change this.

#!/usr/bin/env bash

for d in $(cat /usr/local/etc/prosody/prosody.cfg.lua | grep VirtualHost | sed -r 's/.*"([^"]+)"/\1/'); do
        prosodyctl --root cert import "${d/.conf/}" /usr/local/etc/letsencrypt/live/
done
/usr/sbin/service prosody restart

Get up to date prosody-modules

The one in every distribution repository, including FreeBSD is outdated. You need mercurial aka hg to clone it:

mkdir -p /opt/prosody-modules
cd /opt
hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

Install Prosody

As I mentioned, I'm using FreeBSD, so installing Prosody is simple these days:

pkg install prosody

(optional) luadbi vs SQLite - if you want to use SQLite as backend

In order to have luadbi work with SQLite, it needs to be compiled from ports, because the one from pkg is only built against MySQL. More so, it's built against MySQL 5.7, and so far I failed to built it against MySQL 8.0.

cd/usr/ports/databases/luadbi
make config
You need to compile luadbi to support SQLite3
You need to compile luadbi to support SQLite3
make
make install

Configuring Prosody

Before moving to the prosody config, these probably have to be run:

# these are all ran as the root user
# these are probably not required, they are here just in case
mkdir -p /var/run/prosody
chown prosody:prosody /var/run/prosody
mkdir -p /var/db/prosody
chown prosody:prosody /var/db/prosody

# this is for the http_upload module
mkdir -p /usr/local/www/prosody
chown prosody:prosody /usr/local/www/prosody

# import the letsencrypt certificates using the script from the
# "Letsencrypt post-renewal hook"
# above
bash /usr/local/etc/letsencrypt/renewal-hooks/post/10-prosody.sh

# symlink the main certificate for the legacy SSL services, just in case something needs it
cd /usr/local/etc/prosody/certs
ln -s example.net.crt https.crt
ln -s example.net.key https.key

prosody.lua.cf

plugin_paths = { "/opt/prosody-modules" }
admins = {  "admin@example.net" }

modules_enabled = {
    "announce"; -- Send announcement to all online users
    "blocklist"; -- Allow users to block communications with other users
    "bookmarks"; -- XEP-0048: Bookmarks
    "bosh"; -- Enable BOSH clients, aka "Jabber over HTTP"
    "carbons"; -- Keep multiple clients in sync
    "cloud_notify"; -- XEP-0357: Cloud push notifications
    "csi"; -- client state indication
    "csi_simple"; -- Simple Mobile optimizations
    "csi_battery_saver";
    "dialback"; -- s2s dialback support
    "disco"; -- Service discovery
    "http";
    "http_altconnect";
    "http_files"; -- Serve static files from a directory over HTTP
    "limits"; -- Enable bandwidth limiting for XMPP connections
    "log_auth"; -- Log failed authentication attempts with their IP address
    "mam"; -- XEP-0313, Store messages in an archive and allow users to access it
    "motd"; -- Send a message to users when they log in
    "offline"; --  XEP-0160 XEP-0091, Offline message storage and delayed delivery support
    "pep"; -- XEP-0163, Enables users to publish their mood, activity, playing music and more
    "ping"; --  XEP-0199, Replies to XMPP pings with pongs
    "posix"; -- POSIX functionality, sends server to background, enables syslog, etc.
    "presence"; --
    "private"; -- Private XML storage (for room bookmarks, etc.)
    "proxy65"; -- Enables a file transfer proxy service which clients behind NAT can use
    -- if you are planning to use movim - https://movim.eu/ - you'll need this:
    -- "pubsub"; -- Implements a XEP-0060 pubsub service.
    "register"; -- Allow users to register on this server using a client and change passwords
    "roster"; -- Allow users to have a roster, aka friends list
    "saslauth"; -- Authentication for clients and servers. Recommended if you want to log in.
    "server_contact_info"; -- Publish contact information for this service
    "smacks"; -- XEP-0198: Reliability and fast reconnects for XMPP
    "throttle_presence"; -- presence throttling in CSI
    "time"; -- Let others know the time here on this server
    "tls"; -- Add support for secure TLS on c2s/s2s connections
    "turncredentials"; -- to pass TURN/STUN service that allows voice/video calls in modern clients
    "uptime"; -- XEP-0012, Report how long server has been running
    "vcard4"; -- XEP-0292
    "vcard_legacy"; -- XEP-0398 User Avatar to vCard-Based Avatars Conversion
    "version"; -- Replies to server version requests
    "watchregistrations"; -- Alert admins of registrations
    "websocket"; -- XMPP over WebSockets; useful for web/JS clients
    "welcome"; -- Welcome users who register accounts
}

modules_disabled = {
}

allow_registration = false;
daemonize = true;
pidfile = "/var/run/prosody/prosody.pid";

legacy_ssl_ports = { 5223 }
legacy_ssl_ssl = {
    key = "certs/https.key";
    certificate = "certs/https.crt";
}

c2s_require_encryption = true;
s2s_require_encryption = true;
s2s_secure_auth = true;

auth_append_host = true;

pep_max_items = 10000

-- see https://groups.google.com/g/prosody-users/c/U1LN78jhh_A
network_default_read_size = "*a"

trusted_proxies = { "127.0.0.1", "::1" }

storage = "internal"
-- note: you can, and probably should use an SQL backend
-- but this was not covered in the tutorial. If you were
-- to use MySQL:
--storage = "sql"
--sql = {
--    driver = "MySQL";
--    database = "prosody";
--    host = "localhost";
--    port = 3306;
--    username = "prosody";
--    password = "acbd542f-9009-46e4-9c01-a5a75092160f";
--}
-- or if it's SQLite:
--sql = {
--    driver = "SQLite3";
--    database = "prosody.sqlite";
--}

-- Logging configuration
log = {
    info = "*syslog";
}

-- Certificates
certificates = "certs";

-- BOSH
http_ports = { 5280 }
http_interfaces = { "*" }
https_ports = { 5281 }
https_interfaces = { "*" }
cross_domain_bosh = true
consider_bosh_secure = true
cross_domain_websocket = true

-- Smacks and cloud notify
push_notification_with_body = true
push_notification_with_sender = true
smacks_hibernation_time = 86400

-- Server contact info
-- don't forget to either create an admin@example.net email account
-- or to replace these with the real admin email
contact_info = {
    abuse = { "mailto:admin@example.net", "xmpp:admin@example.net" };
    admin = { "mailto:admin@example.net", "xmpp:admin@example.net" };
    security = { "mailto:admin@example.net", "xmpp:admin@example.net" };
    support = { "mailto:admin@example.net", "xmpp:admin@example.net" };
};

-- Limits
limits = {
    c2s = {
        rate = "100kb/s";
        burst = "2s";
    };
    s2sin = {
        rate = "100kb/s";
        burst = "5s";
    };
}

-- Registration watch
-- so you'll get alerted if anyone registers on your server
registration_watchers = { "admin@example.net" }

-- Proxy
-- this is barely ever used since http_upload, but could come useful
proxy65_ports = { 5000 }
proxy65_address = "YOUR.EXTERNAL.IP.ADDRESS"

-- http_upload
http_upload_path = "/usr/local/www/prosody"
http_upload_file_size_limit = 10485760

-- TURN/STUN
-- for voice/video; it needs another, external service, see later
turncredentials_secret = "CHANGE-THIS-TO-A-LONG-SECRET-WE-WILL-NEED-IT-IN-THE-TURNSERVER";
turncredentials_ttl = 86400;
turncredentials_host = "YOUR.SERVER.FQDN";
turncredentials_port = 3478

-- this is the setting that needs to be turned on to make your server
-- accept registrations
-- without this setting, you can still create users manually,
-- on the server itself.
-- allow_registration = true;

-- Virtual hosts
VirtualHost "example.net"
    -- if you are running dovecot as an imap server on the same host
    -- and it already has authentication configured, use can use this:
    -- authentication = "dovecot"
    -- dovecot_auth_host = "127.0.0.1"
    -- dovecot_auth_port = "9993"
    -- but then you are limited to the users who exist in dovecot.
    -- Alternatively, you can stick to:
    authentication = "internal_hashed"
    -- this is telling the built-in webserver what to really use for URLS
    http_external_url = "https://example.net/"
    http_host = "example.net"

    -- this is the http_upload module to exchange files
    Component "upload.example.net" "http_upload"
        -- some things need to be defined per component, these are one of them:
        http_external_url = "https://example.net/"
        http_host = "example.net"

    -- probably superseeded by http_upload, but could come useful for old clients
    Component "proxy.example.net" "proxy65"

    -- this is to have multi user chats, aka rooms, aka groupchat
    Component "groupchat.example.net" "muc"
        modules_enabled = {
            "muc_mam",  -- For XEP-0313
            "vcard_muc" -- For XEP-0153
        }
   -- If you're using movim - https://movim.eu/
   -- or if you're planning to use this XMPP server as an IoT hub:
   -- Component "pubsub.example.net" "pubsub"
   --    pubsub_max_items = 10000

Start the prosody server

On most systems service prosody start should do.

Don't forget to either create the admin@example.net user, or change the value in the config to one that exists, and can receive alerts:

prosodyctl adduser admin@example.net

Install coturn TURNserver (needed for voice/video)

Again, depending on your system. On FreeBSD:

pkg install coturn

Configure TURN for voice/video

This setup is for a server which is on the internet directly, with a public facing IP address.

turnserver.conf

listening-port=3478
tls-listening-port=5349
listening-ip=0.0.0.0
min-port=49152
max-port=65535
use-auth-secret
static-auth-secret=CHANGE-THIS-TO-A-LONG-SECRET-WE-WILL-NEED-IT-IN-THE-TURNSERVER
server-name=example.net
realm=example.net
cert=/usr/local/etc/prosody/example.net.crt
pkey=/usr/local/etc/prosody/example.net.key
syslog
proc-user=prosody
proc-group=prosody

Open firewall ports

I hope you have a firewall if you have an internet facing server. This is a snippet from my ipfw list

ipfw add 03600 allow tcp from any to me 5222
ipfw add 03700 allow tcp from any to me 5223
ipfw add 03800 allow tcp from any to me 5269
ipfw add 04100 allow tcp from any to me 5000
ipfw add 04700 allow tcp from any to me 3478
ipfw add 04800 allow udp from any to me 3478
ipfw add 04900 allow tcp from any to me 5349
ipfw add 05000 allow udp from any to me 5349
ipfw add 05200 allow udp from any to me 49152-65535
ipfw add 05300 allow tcp from any to me 49152-65535

And in my /etc/rc.conf (note: it contains the TCP port 22, SSH as well, in case you copy-pasted it, to prevent anyone from locking them out from their server. If you don't need it, remove it):

firewall_script="/etc/rc.firewall"
firewall_type="workstation"
firewall_myservices="22/tcp 5222/tcp 5223/tcp 5269/tcp 5000/tcp 3478/tcp 3478/udp 5349/tcp 5349/udp 49152-65535/udp 49152-65535/tcp"
firewall_allowservices="any"
firewall_logging="YES"
firewall_logdeny="YES"
firewall_enable="YES"
firewall_quiet="YES"

Closing words

Running a service is not a simple task, but all of the above is possible to run on a Raspberry Pi, at home, with no extra costs on your existing internet connection - that is if your service provider allows you to run a server. If you have a dynamic IP address, you'll need to be able to change the DNS entries according to it, and fast. DigitalOcean15 offers free DNS service and has an API the allows one to easily alter the needed entries, but it is definitely simpler if you have a fixed IP at home, or if you rent a VPS or a server that has a fixed IP.

For more documentation, https://prosody.im/doc is a wonderful resource, but it's a bit hard to find what you're looking for on it. If you want to test your XMPP server, https://compliance.conversations.im/ offers such a service.

If you spot any mistakes, or have any improvement recommendations, do not hesitate, let me know!

Good luck, and don't forget to spread the word of your artisan service!


  1. https://www.trillian.im/

  2. https://pidgin.im/

  3. https://sourceforge.net/projects/miranda/

  4. https://github.com/petermolnar/awesome-pidgin-plugins

  5. https://xmpp.org/

  6. http://googletalk.blogspot.com/2006/01/xmpp-federation.html

  7. https://www.lifewire.com/set-up-gmail-account-with-macs-mail-application-2260069

  8. https://k9mail.app/documentation/accounts/providerSettings.html

  9. https://www.elliott.org/blog/banned-from-facebook-permanently-how/

  10. https://omemo.top/

  11. https://prosody.im/doc/jingle

  12. https://astrachat.com/

  13. https://www.freebsd.org/

  14. https://movim.eu/

  15. https://www.digitalocean.com/

by Peter Molnar <mail@petermolnar.net>

Ultra low budget photo wall

I made a photo wall on an extremely tight budget, and it doesn't look horrible at all.

Last December we finally moved to our new place - with a 2 month delay due to unforeseen renovation needs. The small room upstairs was always planned to become a study, but then the first lockdowns happened before we got to the point of buying all the needed furniture - so we spent quite a few months waiting for them to reopen, with a lot of boxes everywhere.

When IKEA finally repoened I nearly immediately got there to buy some of their 50cm deep Ivar cabinets - only to learn on unpacking, that unlike the 30cm deep ones, the 50cm ones don't have the crossbar that allowes them to be wall-mounted. Anyhow, ever since then I had a wall, with some cabinets at the feet, with completely empty greyness above it. We opted for a light-mid, tiny bit greenish grey, specifically Valspar Sculpting Clay1 for the walls upstairs; it looks great with natural pine. Valspar is OK if you use either their 700 or Trade graded paints; but avoid the 300 or 500 ones.

So I kept thinking what to do with that empty wall. Initially I was thinking of two larger photos, but I couldn't choose. Then I saw one of the examples on WhiteWall2, but the price of their "Photo Print On Aluminium Backing" was waaaay out of the amount I was willing - or able to - spend on this. Calculating with 24 roughly A4 sized pictures I ended up with, the aluminium option would have been an in the £900 range (24x£39.95) (this is the part when one either bursts into laugh, tears, or coughing), but even the cheapest photo prints would have cost around £200 (24x£9.95 rounded down due to some being smaller, than A4).

We bought a Xerox Phaser 6510 colour laser printer earlier this year to be able to print learning material. It's really very far in quality from professional photo print, but still better, than an lifeless wall.

For mounting, I went for a simple foam board, with the simplest process possible: glue the picture on, cut around with a knife, then use a tiny bit of mounting tape to put it on the wall. I got 4 A1 boards for £10 in Hobbycraft, and in the end, the whole wall costed below £20 (including glue, paper, ink, etc costs).

It's definitely not professional print quality, but it's worth mentioning that it looks a lot better, than the foam board print3 samples I ordered from WhiteWall a long time ago.

My wife help with arranging it: start from the middle. The first ones were the middle 5 pictures, everything else went around them following instincts.

And the result is:

Eastern Suburb Memory Anshun Bridge, Chengdu (closeup) Beddgelert Huanglong Lumsdale Falls Elephant Bathing Pool - Xixiangchi Monastery Dawn at Dojo Stara Wieś Fairy Glen The lake on the rear peak of Mount Qingcheng Hongcun bridge closeup Golden Dragons Nuorilang Waterfall in Jiuzhaigou La Palma - Teneguía Huangshan scenery 3 Sichuan mountains - a view from Jiuzhaigou Primeval Forest Anshun Bridge Sunrise - Hungarian National Graveyard Jiuzhaigou: Peacock Riverbed La Palma - volcano route next to Enríque A view from Barbican Up in the Sky 1 Hills from Beachy Head Snowdonia Llyn Idwal

(Click on the photos to visit them on this site.)


  1. https://www.valsparpaint.co.uk/colours/pre-selected-colours/greys/sculpting-clay/

  2. https://www.whitewall.com/uk/metal-prints/aluminum-mounted-print

  3. https://www.whitewall.com/uk/fine-art-prints/foam-board-printing

by Peter Molnar <mail@petermolnar.net>

Welsh Forest Spirit

license:
Camera
PENTAX K-5 II s
Shutter speed
1/200 sec
Focal length (as set)
85.0 mm
Sensitivity
ISO 400
Lens
K or M Lens

Every once in a while, when in you're in a forest, sit down, and wait. Wait until you start seeing and recognizing the hidden things.

by Peter Molnar <mail@petermolnar.net>

The Troll of the Mine

license:
Camera
PENTAX K-5 II s
Shutter speed
1/400 sec
Focal length (as set)
50.0 mm
Sensitivity
ISO 80
Lens
K or M Lens

A rock troll, guarding the Parys copper mine.

by Peter Molnar <mail@petermolnar.net>

Parys Copper Mines

license:
Camera
PENTAX K-5 II s
Aperture
f/8.0
Shutter speed
1/250 sec
Focal length (as set)
35.0 mm
Sensitivity
ISO 80
Lens
HD PENTAX-DA 16-85mm F3.5-5.6 ED DC WR

On the edge of Anglesey, there's an abandoned copper mine; a landscape, shaped from the end of the 18th century till the middle of the 20th.

by Peter Molnar <mail@petermolnar.net>

Sheep dotted landscape

license:
Camera
PENTAX K-5 II s
Aperture
f/8.0
Shutter speed
1/400 sec
Focal length (as set)
85.0 mm
Sensitivity
ISO 80
Lens
HD PENTAX-DA 16-85mm F3.5-5.6 ED DC WR

Another lake, another hillside; but this time, with plethora of sheep. And insane lights, but that's normal for Wales.

by Peter Molnar <mail@petermolnar.net>

A view from Idwal

license:
Camera
PENTAX K-5 II s

This isn't the first image I took from the same scenery, but this is the first panorama that is acceptable. At the Lake Idwal parking lot, there's a small road that leads into the mountains; the view is much better for a panorama, from there, than from the main road.

by Peter Molnar <mail@petermolnar.net>

Cwmortin House

license:
Camera
PENTAX K-5 II s
Aperture
f/8.0
Shutter speed
1/250 sec
Focal length (as set)
35.0 mm
Sensitivity
ISO 100
Lens
smc PENTAX-DA 35mm F2.4 AL

This is the aforementioned lake from the previous post. I'm rather sure the white house it not abandoned: all of it's windows had a wildlife watch camera.

by Peter Molnar <mail@petermolnar.net>

Capel Rhosydd

license:
Camera
PENTAX K-5 II s
Aperture
f/5.0
Shutter speed
1/500 sec
Focal length (as set)
87.5 mm
Sensitivity
ISO 100
Lens
Tamron SP AF 70-200mm F2.8 Di LD [IF] Macro (A001)

Always talk to the locals; especially when they have a lovely, very friendly dog, from whom there's no escape. Long story short: we were recommended that once we're up at the lake, take the bridge and the trail that goes next to the abandoned building you see on the picture - it is supposed to lead to a whole abandoned mining town. Given the weather - misty, with low hanging clouds - it would have been a hell of an experience.

Unfortunately, we couldn't do this: there was no bridge. So much rain had fallen there recently, that the bridges were literally washed away, or sunked into the lake.

by Peter Molnar <mail@petermolnar.net>

Excavating my former homepages

Someone, a long time ago came up with the thought that whatever is put on the internet, it'll be there forever. Well, it's wrong. The old versions of my own website, including their design, were long gone, so I decided to put the Indiana Jones hat on, and started digging.

It's early 1999. I'm 13 years old, just entered high school1. Altavista is still a decent search engine. At this point I've been using the web for years, but never really participated in making it yet, and it was about to change. The idea of having a homepage stuck with me: I could make my own design, my own little world, without limits, with any kind of content I want to, without anyone supervising. This was such an exciting prospect that it had to be done. Plus: it was open for anyone, like instantly hanging your work in an art gallery!

Long story short, I did that website. And the next one, and the next one... and I kept going till this very day. The tragedy of it is that due to multiple reasons, the first two iterations and designs of my site are now completely lost.

Back in those days my English skills or my "programming" skills were quite lacking. I never heard the idea of version controlling, digital archiving, snapshots - these came so much later. Hungary tends to teach foreign languages with an overwhelming amount of grammar, resulting in a nice, but unusable skill when someone wants to explore the internet.

Put these and the options on the mid 90s web together - generic Hungary is always around 5 years behind the English speaking world in internet trends. My first sites were made with Microsoft FrontPage 98, FTP-d directly into "production", always overwriting the tiny, free space on free homepage providers, hopping from one to another, because the new one offered 5MB for free, not just 1.

1999 - 2003 - the FrontPage years

When I decided to go after the old versions of my gazillion URLs, I first turned to archive.org, like everyone would. For my gigantic surprise, after remembering that the provider extra.hu had subfolders, and not sub-domains, I found a version from May 2001! A year earlier than my earliest archive! Sadly, it was the already on it's third design, and because archive.org didn't have the images from back then, it wouldn't have mattered anyway.

The good news was that my site at this point was static HTML, so "recovering" it was simply opening the files. Did you know web browsers still support frames?

I went through this version quite a few times lately, and I always overlooked a section, in which I had a review of an Ericsson T29s - including notes on how to hack a data cable for it, downloadable programs, full eeprom and flash binaries, and a fascinating note of:

I'm utterly tired that there are only ads on every page, and no real stuff, so I thought I'll put some here, based on my own experience

Again, this is roughly 2002. It's good to be reminded by my own words, that the internet had this problem for a very long time by now.

I decided put the review back up - look under the IT tab -, even though chances that someone is looking for these is converging to 0.

2002 design - tables, absolutely no accessibility, but hey, the
content was already about tinkering with phones, and the colour palette
looked very similar to my current dark theme.
2002 design - tables, absolutely no accessibility, but hey, the content was already about tinkering with phones, and the colour palette looked very similar to my current dark theme.

2004 - the PHP4 years

To access the 2004 version, magic has to be called upon - also known as virtual machines. I had to spin up a Debian Sarge2, because I needed PHP4 - immeasurable kudos for the Debian archives3 for still having install CDs, and working software repositories! The only trick is that during the install the apt mirror needs to be set to archive.debian.org.

I barely remembered this version. It also made me realise that I was microblogging, before it even became a term - although as Kevin Marks4 pointed it out on the indieweb-chat channel on Freenode, everyone was microblogging at first: the idea of one page per entry came later.

After not too much digging I found two more phone reviews, also with a lot of background content - in case you're after an 50KB mobile Java email application, I have one for you.

2004 - the layout probably worked in IE5 and nothing else
2004 - the layout probably worked in IE5 and nothing else

Instead of webrings, in Hungary, we regularly exchanged banners with each other: small images, that linked to another site; sort of a graphical blogroll. Contrary to mathematical probability, I found a banner that is still alive! 16 years and still strong: the Hungarian Stargate fan site5 deserves some attribution for that banner6

2007 - 2009 - the custom CMS with friends years

2007 brought a massive change: I wanted to make the site tiny a community site, a place to where my close friends could also upload photos, short stories, art.

3 years meant PHP5, so I spun a Debian Etch7 up. These were the days when I had my custom, PHP-based CMS, of which if I looked at the code of; it's proper nightmare material.

The revelation of the 2007 version was the amount of photos I sort of accidentally removed from my site. In the coming years my site slowly morphed into a portfolio for my photos and my sysadmining skills. When I finally reverted, and swapped from curated, small galleries to a stream of individual images, I never put the old portfolio back. This will need to be addressed.

Related to this, I came across Ana's "Blogging and me" post at https://ohhelloana.blog/blogging-and-me/ . She went through very similar cycles as I did: a website first, blogging of anything, that gradually became a hyper-focused, work-only site, which then got neglected, because it wasn't fun any more. Read it. If you didn't (yet) had these cycles, or if you already did; in both cases, it's a fantastic piece.

Such design, much images. Also, curves! Curves everywhere! Regardless,
I believe this was a pleasent to look at design and colour combination,
plus those photos are all missing from my site at the moment
o.O
Such design, much images. Also, curves! Curves everywhere! Regardless, I believe this was a pleasent to look at design and colour combination, plus those photos are all missing from my site at the moment o.O

2008 - 2013 - the portfolio years

Parallel to the community site idea, I wanted to have a portfolio site; one that is professional content only. At that point, I still believed my life will have something to do with photography, so I made a photo portfolio site under the new domain petermolnar.eu. My name is quite a common one in Hungary, but the .eu domains were new, and I grabbed it immediately.

During these years I actively participated in an alternative community in Hungary. I wrote some articles to their site, which are still up, to this very day, but the idea to put those on my own site (as well) didn't occur to me.

In 2010 , the limitations of my CMS collided with the fact that I lost touch with most people who had some content on that community site. I decided to close it, and turn my full attention towards a personal site

Soon after college (2009), I ended up working as a sysadmin; it made sense to start writing my findings, my how-tos (mostly for myself) down.

With that, I've thrown a silly amount of old content away. No thoughts or personal blog entries any more: streamlined photo portfolio, photo equipment review - this one never even took off; the 2 entries I wrote for it are so low quality that they were not worth salvaging -, and a sysadmin blog is the way to go! I should never have done that, but these were the early Facebook years; everyone was doing personal communication in social silos. It seemed like a good idea that time. Mea culpa.

I did try cross-posting and syndicating early on: links to new entries to twitter, to facebook, but soon it looked overwhelming. When the same content goes everywhere, and you happened to have a large cross-section of the same people on each silo, it'll be too much for them.

First iteration of my very own WordPress theme - along with the
immeasurable changes of finally moving to my own domain, with only
professional-ish content
First iteration of my very own WordPress theme - along with the immeasurable changes of finally moving to my own domain, with only professional-ish content

Slowly, but steadily, topics and sections kept creeping back in. At first, it was merely a photoblog tab, in which I put collection of images grouped by a topic. It took me a while to realise I didn't like it that way: without context, the galleries were distant; they didn't feel like memories, more like and extended, endless portfolio, yet I kept going with it for years.

Second WordPress theme from somewhere in 2011
Second WordPress theme from somewhere in 2011

By the end of 2013, I was back to having a "world view" section that was to become "journal". The trigger for this was our relocation to England: there was an irrepressible urge to write about our new life.

Thankfully the streamlining didn't stick and the site kept getting
more content sections back
Thankfully the streamlining didn't stick and the site kept getting more content sections back

2014 - 2017 - The features and contents years

Sometimes in 2014 I stumbled upon the indieweb8 community. With that, my site started to get features - and a lot of them.

2014

See those bubbles? Those were mainly comments, backfilled from
Facebook.
See those bubbles? Those were mainly comments, backfilled from Facebook.
Calculated read time; I believe medium.com brought it in.
Calculated read time; I believe medium.com brought it in.
I still had my photo portfolio as well. It was already a responsive
design; on mobile, it became a simple list of full-width
images.
I still had my photo portfolio as well. It was already a responsive design; on mobile, it became a simple list of full-width images.

Once you read too many studies on the internet you may start questioning your own experiences with certain things. One of these is the "light on dark" vs "dark on light" representation when it comes to computers and the web. To obey this, I started presenting text content in light, photo content with dark background and themeing; a route I should never have taken.

During the 15 years prior to this, my site was dark. This page is supposed to be my home, and it's supposed to represent me on the internet. And I'm not a light background person; I never was. When I still used Window XP it looked like this:

The bottom is a launcher bar, not opened
applications.
The bottom is a launcher bar, not opened applications.

Even so, for the coming years, the design went lighter and lighter.

2016

Painfully bright, and never felt like my own. Plus there was a huge
stream of content: bookmarks, quotes, twitter reply copies, etc. I
turned my site into a stream, and the more content it had, the more
distant it felt, the less of a homepage it became.
Painfully bright, and never felt like my own. Plus there was a huge stream of content: bookmarks, quotes, twitter reply copies, etc. I turned my site into a stream, and the more content it had, the more distant it felt, the less of a homepage it became.

2017

In 2017, I found the icons I'm using till this very day, but I still had the dreaded light design. Plus I added even more content, including the idea to backfill EVERYTHING I ever made on the internet. Needless to say this idea was later reverted, though without the journey, I may never have realized why it was bad.
In 2017, I found the icons I'm using till this very day, but I still had the dreaded light design. Plus I added even more content, including the idea to backfill EVERYTHING9 I ever made on the internet. Needless to say this idea was later reverted, though without the journey, I may never have realized why it was bad.

2018 - back to the roots: dark, static, simple.

Content was dropped10, and I again, felt like I have a homepage, and not a one person social media site.

Back to the darkness. Still not the best, but much less strain on my
eyes.
Back to the darkness. Still not the best, but much less strain on my eyes.
Turned out the studies were right about a few things, and that for
some, light on dark is unreadable. So for everyone to be happy, I added
an option to switch between them, making the dark, or the operating
system setting the default. I also tried to make it look like some
really messed up terminal. It didn't work.
Turned out the studies were right about a few things, and that for some, light on dark is unreadable. So for everyone to be happy, I added an option to switch between them, making the dark, or the operating system setting the default. I also tried to make it look like some really messed up terminal. It didn't work.

And thus we arrived to 2020, and you're looking at the freshest, most current iteration of my website.

Conclusions

I think at the beginning I started to dig my sites up to verify how much have I gone back to my roots. To see how it started, how it done, when it was solely for the fun. Ana's post, which I already mentioned11 was definitely a trigger to take this journey.

I lost the first two designs. I remember the very first vividly, with the water-like repeating background. The second I completely forgot about, until I found a reminder of my proto-microblog entries from 2001

My conclusions are simple: your homepage is called that for a reason. It's a poster, a gallery, a window to you, out for anyone to visit, to see. It's not a social media profile. It's not a resume. It's you.

Whether you want it to be a portfolio, a blog, it doesn't matter; just make it sure it represents you, and that you keep liking it, and keep enjoying working on it, extending it.


  1. https://en.wikipedia.org/wiki/Education_in_Hungary

  2. https://cdimage.debian.org/mirror/cdimage/archive/3.0_r6/i386/iso-cd/

  3. https://cdimage.debian.org/mirror/cdimage/archive/

  4. http://www.kevinmarks.com/

  5. http://www.csillagkapu.hu/

  6. http://www.csillagkapu.hu/img/agbanner.gif

  7. https://cdimage.debian.org/mirror/cdimage/archive/4.0_r0/i386/iso-cd/

  8. https://indieweb.org/

  9. https://petermolnar.net/article/personal-website-as-archiving-vault/index.html

  10. https://petermolnar.net/article/making-things-private/index.html

  11. https://ohhelloana.blog/blogging-and-me/

by Peter Molnar <mail@petermolnar.net>