What’s new in PyLink – May 26, 2018

A busy first year of university has finally concluded… Now, here is a look at what’s been happening in PyLink development over the last half year. To keep this post at a manageable length, I will only be focusing on the new stuff released this month. (The development branch meanwhile has gotten more updates since than just that; you can find the interesting bits of 2.0-alpha1 and 2.0-alpha2 in their release notes.)

2 releases in May 2018

Two new releases were pushed this month, 1.3.0 in the stable series and 2.0-alpha3 in the development branch.

PyLink 1.3.0

The 1.3 series focuses on backporting some commonly requested features from the 2.0 (devel) branch.

So far, this has included a backport of the 2.0 launcher, bringing in daemonization support, the ability to ignore stale PID files, and shutdown/restart/rehash support via the command line.

As well, 1.3 adds the ability to restrict login blocks to opered users, specific networks, and specific user hosts. Service bot fields and relay server suffix can now be set by network, and relay gained support for a few more modes such as InspIRCd blockhighlight +V and exemptchanops +X.

These changes are documented more precisely in the 1.3.0 release notes.

PyLink 2.0-alpha3

2.0-alpha3 was not a massive release feature-wise, though some highlights include it being the first 2.0 snapshot to include the daemonization support now ported to in 1.3, as well as a new (alpha-quality) antispam plugin specifically targeting mass-highlight spam. Other minor details include Clientbot now supporting expansions such as $nick in autoperform, and better STATUSMSG handling in relay.

This release instead focused on reworking several parts of PyLink’s core, including sockets, service bots, mode handling, and user tracking. I will list some of the major bits here:

Porting PyLink to select

In 2.0-alpha3, PyLink’s socket handler was ported to use select instead of one listener thread per network. (More specifically, it now uses the selectors module in Python 3.4+, which is a higher-level frontend to epoll/poll/select/etc.)

Because this port is still somewhat new, I have noticed some instability when it comes to handling disconnects. In particular, there are some sporadic issues with autoreconnect not firing that still need to be investigated. That said, I cannot say if the 2.0 stack is more or less stable than the 1.3 one since I have been running 2.0 snapshots on my production instance for quite some time. (As always, testers are welcome!)

Reworking channel handling in service bot handling (API break)

Prior to this release, service bot channel handling and relay were a complete mess. Referring back to issue#265, PyLink 1.x had sore points where service bots wouldn’t rejoin relay leaf channels on kill and lingered around when a channel was delinked. Later 2.0 snapshots had the opposite issue where service bots left the main (hub) channel when any leaf network was delinked.

After some thought, my solution to these problems was to make the entire notion of “plugins making service bots join channels” a plugin-specific mechanism. In the end, 2.0-alpha3 introduced several new functions: ServiceBot.add_persistent_channel(), ServiceBot.remove_persistent_channel(), ServiceBot.get_persistent_channels() which use namespaces to track which channels different plugins want a service bot to be in.

ServiceBot.join() was repurposed to explicitly join service bots to channels (without remembering them as persistent), while ServiceBot.part() was added to trigger a part request, where a service bot would only actually leave a channel after no plugin marked it as persistent. Kill and kick handling in services_support were similarly reworked to use the new get_persistent_channels(), which also adds in config-defined service bot channels to any set by plugins.

Another somewhat experimental change is that service bots now try to be dynamic: that is, they will leave persistent channels once they become empty and rejoin them whenever someone else does. No considerations have been made yet to restore channel timestamps and reverse channel takeovers etc., since channel protection is somewhat out of our scope as an extended service.

Fixing edge cases in mode parsing (issue#573)

In PyLink versions before 2.0-alpha3, attempts to prevent modesetting loops  caused an issue where mode changes that both set and unset a particular mode dropped the unsetting portion when parsing. That is, if you tried to set +b-b *!*@test.host *!*@test.host or something similar, plugins like relay would see it as +b *!*@test.host only and leave you with a pretty confusing desync.

This problem was fixed in 2.0-alpha3 by reworking the mode parser to apply mode changes incrementally (onto a temporary mode state) as each single mode change (or internally, “mode pair”†) was read. This allows the check that prevents removing nonexistent modes to always have an up to date reference when it decides whether to keep or drop a specific mode change.

Work still needs to be done to potentially backport this to 1.x, though doing so is somewhat difficult because the changes are invasive (and because core function names in 1.x and 2.x no longer match up).

† Mode pairs are how mode strings in PyLink (and other projects like Supybot/Limnoria) are internally represented. The format is simple, with individual mode such as +o james split into a 2-item tuple: ("+o", "james††"). If a mode takes no argument, the second item is None (null).

†† Actually, PyLink would store the argument for a status mode change as a UID identifier†††, which stays consistent through nick changes and the like.

††† When this isn’t possible, numbered “Pseudo-UIDs††††” are used instead.

†††† Just kidding, there are enough footnotes here.

Seriously optimized user tracking

This alpha snapshot brought about significant optimizations to how users are tracked. Previously, each network’s users index only contained only a single mapping from user IDs to nicks, while going in reverse (nick→UID) required a reverse lookup through the entire user list every single time. This was a giant misdesign on my part because it simply did not scale for networks going up to a few hundred users.

Fortunately, I realized a quick and elegant fix by overwriting the setter for the each user object’s ‘nick’ attribute to write to a nick→UID mapping for us. Eventually, that became irc.users.bynick, which tracks UIDs by normalized (lowercased) nicks and is completely abstracted away from PyLink protocol modules.

Leave a Comment