vndb/manned
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Large-ish SQL schema revamp/optimizations

Browse source 
Primarily aimed at reducing the size of the old 'man' (now: files)
table, using smaller integers to refer to man contents and text fields,
and storing a shorthash as an integer for quick lookups. This better
normalization also removes the need to keep a separate 'man_index' cache
for the search function.

The old schema wasn't necessarily bad, but I was in the mood for some
optimizations. And a little cleanup.

Prolly introduces a bunch of new bugs, I haven't tested this too well.
This commit is contained in:
Yorhel 2021-12-14 15:06:05 +01:00
parent 6f7f59c6df
commit f376f1f137
6 changed files with 268 additions and 128 deletions
Download patch file Download diff file Expand all files Collapse all files

8
util/cron.sh
View file

@ -10,9 +10,5 @@ PSQL="psql -U manned -Awtq"
./fedora.sh current
./ubuntu.sh current
# Only update indices once a week (on mondays). This process is slow and the data doesn't often change anyway.
if [ `date +%u` == 1 ]
then
echo "============ Updating SQL indices"
$PSQL -f update_indices.sql
fi
echo "============ Updating SQL indices"
$PSQL -f update_indices.sql

12
util/update_indices.sql
View file

@ -2,14 +2,10 @@
-- the table being replaced. The site should remain responsive while these
-- queries are run.
BEGIN;
CREATE TABLE man_index_new AS SELECT DISTINCT name, section FROM man;
CREATE INDEX ON man_index_new USING btree(lower(name) text_pattern_ops);
DROP TABLE man_index;
ALTER TABLE man_index_new RENAME TO man_index;
COMMIT;
BEGIN;
CREATE TABLE stats_cache_new AS SELECT count(distinct hash) AS hashes, count(distinct name) AS mans, count(*) AS files, count(distinct package) AS packages FROM man;
CREATE TABLE stats_cache_new AS
SELECT (SELECT count(*) FROM contents) AS hashes,
(SELECT count(distinct name) FROM mans) AS mans, *
FROM (SELECT count(*), count(distinct pkgver) FROM files) x(files, packages);
DROP TABLE stats_cache;
ALTER TABLE stats_cache_new RENAME TO stats_cache;
COMMIT;