The frontend always stripped off the encodings already, so no point in
keeping that in the DB indices. The full locale was extracted from the
filename, which we still keep, so no information is list.
SQL "migration" script:
BEGIN;
CREATE INDEX files_tmp_locale ON files (locale);
INSERT INTO locales (locale) VALUES ('pl_PL'), ('is_IS'), ('ko_KR');
WITH obs(id, locale, lang) AS (
SELECT id, locale, regexp_replace(locale, '^([^.]+)\..+$', '\1') FROM locales WHERE locale LIKE '%.%'
UNION ALL
SELECT id, locale, '' FROM locales WHERE locale LIKE 'node%' OR locale = 'common'
), rep(old, new) AS (
SELECT o.id, x.id FROM obs o LEFT JOIN locales x ON x.locale = o.lang
), upd AS (
UPDATE files SET locale = new FROM rep WHERE locale = old
) DELETE FROM locales WHERE id IN(SELECT id FROM obs);
DROP INDEX files_tmp_locale;
COMMIT;
Going from average ~100ms to ~10ms or so. The previous query had a
tendency to be much slower sometimes, let's see if this cache also takes
care of those outliers.
Migration script:
ALTER TABLE packages ADD COLUMN c_hasman boolean NOT NULL DEFAULT FALSE;
DROP INDEX packages_system_name_key;
CREATE UNIQUE INDEX packages_system_name_key ON packages (system, name) INCLUDE (id, c_hasman, dead);
UPDATE packages SET c_hasman = NOT c_hasman
WHERE c_hasman <> EXISTS(SELECT 1 FROM package_versions pv WHERE pv.package = packages.id AND EXISTS(SELECT 1 FROM files f WHERE f.pkgver = pv.id));
Which I broke in 83ab6c3671. Need to find
an alternative approach to detecting dead packages sometime. The 'dead'
flag isn't super important so it can wait.
This provides an almost 2x speedup in man page rendering time and
removes some heuristics to work around bad guesses by grog(1).
Funnily enough, this also fixes rendering of obscure man pages that
happen to use 'grap' macros; grog detected those correctly but my groff
installation doesn't actually support rendering that.
No doubt I broke rendering of other pages, will have to see.
Whether or not the package name itself or the (category,name) tuple
uniquely identified a package within a system has been a source of
confusion for a long time. Back in
03d278e4ff I ended up playing playing it
"safe" by going for (category,name), but in practice this doesn't make a
whole lot of sense. While it's *possible* for the same package name to
refer to completely different packages in different "categories", in
reality distributions can't sanely support this anyway.
For distributions where the category referred to a repository, the only
cases where the same package name was used in different repos was when
the package has moved from one repo to another. Those should certainly
not be treated as different packages.
For distributions where the category really referred to a category,
there's the Debian approach where the category is purely a tag and
doesn't help identify the package in any way, and then there's FreeBSD
where the category technically ought to be part of the name. There were
a few cases where FreeBSD used categories to separate out different
versions of the same package (e.g. ipv6 vs non-ipv6), but none were
relevant for man pages so I ended up merging those as well.
Getting rid of the categories simplifies and shortens URLs, unclutters
the UI a little bit and merges the packages in listings that should've
been merged all along.
Migration script:
-- Merge packages that are in multiple categories.
-- All versions are moved to the package with the lowest ID.
-- If the same version already exists in a lower ID, the higher-ID version is deleted.
BEGIN;
WITH migrate(old, new, second) AS (
SELECT q.id, MIN(p.id), MAX(p.id)
FROM packages p
JOIN packages q ON q.id > p.id AND p.system = q.system AND p.name = q.name
GROUP BY q.id
), ded(n) AS (
UPDATE packages SET dead = false
FROM migrate m
JOIN packages q ON q.id = m.old
WHERE packages.id = m.new AND packages.dead AND NOT q.dead
RETURNING 1
), mov(n) AS (
UPDATE package_versions SET package = m.new
FROM migrate m
WHERE package_versions.package = m.old
AND NOT EXISTS(
SELECT 1
FROM package_versions v
WHERE v.package IN(m.new, m.second)
AND v.version = package_versions.version)
RETURNING 1
), del(n) AS (
DELETE FROM packages WHERE id IN(SELECT old FROM migrate)
RETURNING 1
) SELECT (SELECT count(*) FROM migrate) AS migrate,
(SELECT count(*) FROM ded) AS ded,
(SELECT count(*) FROM mov) AS mov,
(SELECT count(*) FROM del) AS del;
ALTER TABLE packages DROP CONSTRAINT packages_system_name_category_key;
CREATE UNIQUE INDEX packages_system_name_key ON packages (system, name);
ALTER TABLE packages DROP COLUMN category;
COMMIT;
Removing the JS requirement and (hopefully) providing a more useful
view into the same data.
This view now also lists other man pages that happen to have the same
contents.
This fixes selecting the right man page for 'mount', which would
otherwise grab a version from an old distro that happened to have it
with an explicit 'en' locale.
New canonical permalink format is now:
/man.<hash>/<system>/<name>.<section>
Including the system name makes for more predictable navigation between
man pages of the same system.
This new URL format also fixes a problem with not being able to browse
from a package's page to the correct page when the package has multiple
versions of the same man.
This is also yak shaving for some new navigation and formatting features
I have planned.
contact@manned.org works fine, too, except I don't have outgoing mail
setup to go through an smtp proxy so I can't reply to Outlook and gmail
(and a few others). Seriously, fuck providers who can't see the
difference between spam messages and human replies to legitimate mails.
The /man/* URLs now directly open man pages rather than redirecting to
the permalink format, so that the extra source information can be used
to provide a better UI for switching between systems and packages
(currently only between systems).
Step one in getting rid of the JS location/version selection thingies.
Still quite a bit more to do, but I haven't worked it all out yet.
Primarily aimed at reducing the size of the old 'man' (now: files)
table, using smaller integers to refer to man contents and text fields,
and storing a shorthash as an integer for quick lookups. This better
normalization also removes the need to keep a separate 'man_index' cache
for the search function.
The old schema wasn't necessarily bad, but I was in the mood for some
optimizations. And a little cleanup.
Prolly introduces a bunch of new bugs, I haven't tested this too well.
Also greatly simplified basename_from_filename() because apparently I
couldn't write regexes back then.
(And the removed REFERENCES line is to sync schema.sql with the actual
state of the DB, which doesn't have that constraint for some reason.
I'll prolly fix that later)
Not that I have anything against the MIT license nowadays, nor do I mind
of companies use my software without paying and all that stuff, but this
switch is because I *do* like the AGPL license more for networked
software like this, especially when getting-everyone-and-their-pet to
run the code is not really a goal.
Of course, previous commits are still available under MIT.
We've got a lot of packages in the DB that have long been removed from
the Arch repos. These are still indexed, but won't clutter the package
listing anymore.
Also fixed an issue with packages.id numbers getting rather large
because the indexer allocates a new ID for every package on every
update.
I didn't touch the actual man page pages yet, but they seem to have
mostly survived my CSS changes. There's a LOT of general UI improvements
to be done, while this commit is really just a different style of paint.
