Lots of changes

- ncdc 1.6 released
- ncdc FAQ updates
- 3 new images for /dump/nccol
- added my final report to /doc
- added ncdc-transfer-stats and ncdc-share-report to /dump

dat/ncdc-changelog

@@ -1,3 +1,25 @@
+1.6 - 2011-12-07
+	- Use SQLite3 for storage instead of GDBM
+	- Converted config.ini to SQLite3 database
+	- Added ncdc-db-upgrade utility
+	- Session directory is architecture-independent
+	- All data is safe against crashes and power failures
+	- Added support for removing/adding directories without rehashing
+	- Always match every file list on 'Q' key on TTH search
+	- Immediately flush log entries to the kernel
+	- Faster start-up
+	- Added support for per-hub 'active_ip' settings
+	- Allow interval notation when setting autorefresh
+	- Broadcast SF (number of shared files) on ADC hubs
+	- Combine TTH data for downloaded files to blocks of at least 1MiB
+	- Increased hash buffer size (10KiB -> 512KiB)
+	- Fix case-insensitivity of search results
+	- Fix reporting of user state in pm tabs at hub disconnect
+	- Fix generation of client certificates with openssl
+	- Fix segfault with duplicate users on an ADC hub
+	- Fix segfault when opening of a filelist fails
+	- Fix base32 decoding bug (fixes login sequence on some ADC hubs)
+
 1.5 - 2011-11-03
 	- Added filelist_maxage setting
 	- Added flush_file_cache setting
@@ -10,7 +32,8 @@
 
 1.4 - 2011-10-26
 	- Added sorting functionality to file list
-  - Added color settings: title, separator, list_default, list_header and list_select
+	- Added color settings: title, separator, list_default, list_header and
+	  list_select
 	- Added "blink" color attribute
 	- Allow /disconnect to be used on the main tab
 	- Display number of matched and added items when using match queue feature
@@ -129,7 +152,7 @@
 	- Fixed segfault when the userlist is open while disconnecting from a hub
 
 0.4 - 2011-07-23
-	- Added file downloading support.
+	- Added file downloading support
 	  WARNING: Downloaded files are not TTH checked at this moment.
 	- Added persistent download queue
 	- Added busy indicators on start-up and with /gc

dat/ncdc-faq

@@ -50,19 +50,26 @@ up-to-date client that still uses any of these features.
 This is for Ubuntu 11.10, but may work for other versions as well. Run the
 following command:
 
-  sudo apt-get install libbz2-dev libgdbm-dev\
-    libncursesw5-dev libxml2-dev libglib2.0-dev gnutls-bin
+  sudo apt-get install libbz2-dev libsqlite3-dev\
+    libncursesw5-dev libxml2-dev libglib2.0-dev
 
 And then follow the instructions in the README.
 
 
-=head2 I'm getting the error "No PEM-encoded private key found" on startup
+=head2 Ncdc uses too much disk space!
 
-Most likely this is caused by a L<bug in
-glib-networking|https://bugzilla.gnome.org/show_bug.cgi?id=664321>. To get
-around it, install the "certtool" utility that comes with gnutls (package
-"gnutls-bin" on Ubuntu), delete the old certificates (C<rm ~/.ncdc/cert/*>),
-and then start ncdc again.
+First, look where this disk space goes to (hint: use
+L<ncdu|http://dev.yorhel.nl/ncdu>). If it's the log files: you can safely
+delete or rotate them (see next question).
+
+The I<db.sqlite3> file can also grow quite large in certain situations. If you
+modify or rename a lot of files in your share and ncdc re-hashes them, the old
+hash data associated with the files is not removed from the database, resulting
+in wasted disk space. The C</gc> command in ncdc can be used to clean up this
+unused data. Be warned, however, that this command needs roughly twice the size
+of the old db.sqlite3 file for temporary storage, so make sure you have enough
+space available. (Note that this behaviour is not specific to ncdc, most other
+DC clients do the same.)
 
 
 =head2 Why doesn't ncdc rotate log files automatically?
@@ -73,6 +80,15 @@ L<this is the script I use|http://p.blicky.net/s7132>, which is run as a
 monthly cron job.
 
 
+=head2 I'm getting the error "No PEM-encoded private key found" on startup
+
+(This issue should have been fixed in 1.6) Most likely this is caused by a
+L<bug in glib-networking|https://bugzilla.gnome.org/show_bug.cgi?id=664321>. To
+get around it, install the "certtool" utility that comes with gnutls (package
+"gnutls-bin" on Ubuntu), delete the old certificates (C<rm ~/.ncdc/cert/*>),
+and then start ncdc again.
+
+
 
 =head1 Can ncdc...
 
@@ -123,6 +139,7 @@ myself, though. I just run ncdc directly on my router. :-)
 
 =head2 Are there any programs available for analyzing the transfers.log file?
 
-Nothing like that is included in the release yet, but there is a simple
-Perl script available: L<ncdc-transfer-stats|http://p.blicky.net/agolr>.
+Nothing like that is included in the release yet, but there is a simple Perl
+script available: L<ncdc-transfer-stats|http://p.blicky.net/agolr>, and a short
+Go program: L<ncdc-share-report|http://p.blicky.net/6yx2d>.
 

dat/ncdc-man

@@ -38,13 +38,13 @@ GETTING STARTED
          /set active true
 
        See the help text for each of the commands and settings  for  more  information.
-       Of  course,  all  of  the above settings are saved to the configuration file and
-       will be used again on the next run.
+       Of  course, all of the above settings are saved to the database and will be used
+       again on the next run.
 
        To connect to a hub, use /open:
          /open ncdc adc://dc.blicky.net:2780/
        Here `ncdc' is the personal name you give to the hub, and  the  second  argument
-       the URL. This URL will be saved in the config file, so the next time you want to
+       the  URL.  This  URL will be saved in the database, so the next time you want to
        connect to this hub, you can simply do:
          /open ncdc
        See the help text for '/open' and '/connect' for more information. If  you  want
@@ -88,7 +88,7 @@ INTERACTIVE COMMANDS
        /accept
               Use this command to accept the TLS certificate of a hub. This command  is
               used  only  in the case the keyprint of the TLS certificate of a hub does
-              not match the keyprint stored in the configuration file.
+              not match the keyprint stored in the database.
 
        /browse [[-f] <user>]
               Without arguments, this opens a new tab where you  can  browse  your  own
@@ -133,11 +133,16 @@ INTERACTIVE COMMANDS
 
        /gc
               Cleans up unused data and reorganizes existing data to allow  more  effi‐
-              cient  storage and usage. Currently, this commands cleans up hashdata.dat
-              and dl.dat, removes unused files in inc/ and old files in fl/.
+              cient  storage  and  usage.  Currently, this commands removes unused hash
+              data, does a VACUUM on db.sqlite3, removes unused files in inc/  and  old
+              files in fl/.
 
               This  command  may  take some time to complete, and will fully block ncdc
-              while it is running. You won't have to perform this command very often.
+              while it is running. It is recommended to run this command every once  in
+              a while. Every month is a good interval. Note that when ncdc says that it
+              has completed this command, it's lying to you. Ncdc will still run a  few
+              large  queries  on  the background, which may take up to a minute to com‐
+              plete.
 
        /grant [-list|<user>]
               Grant someone a slot. This allows the user to download from you  even  if
@@ -194,7 +199,7 @@ INTERACTIVE COMMANDS
 
        /password <password>
               This  command can be used to send a password to the hub without saving it
-              to  the config file. If you wish to login automatically without having to
+              to the database. If you wish to login  automatically  without  having  to
               type  /password  every  time,  use '/set password <password>'. Be warned,
               however, that your password will be saved unencrypted in that case.
 
@@ -286,9 +291,11 @@ INTERACTIVE COMMANDS
               To remove a single directory from your share, use `/unshare  <name>',  to
               remove all directories from your share, use `/unshare /'.
 
-              Note that all hash data for the removed directories will be thrown  away.
-              All files will have to be re-hashed again when the directory is later re-
-              added.
+              Note  that the hash data associated with the removed files will remain in
+              the database. This allows you to re-add the files to your  share  without
+              needing  to re-hash them. The downside is that the database file may grow
+              fairly large with unneeded information. See the `/gc'  command  to  clean
+              that up.
 
        /userlist
               Opens  the  user list of the currently selected hub. Can also be accessed
@@ -304,7 +311,7 @@ INTERACTIVE COMMANDS
 
 SETTINGS
        The following is a list of configuration settings. These settings can be changed
-       and queried using the `/set' command, and are stored in the config.ini  file  in
+       and  queried  using the `/set' command, and are stored in the db.sqlite3 file in
        the session directory.
 
        active <boolean>
@@ -321,11 +328,9 @@ SETTINGS
               the internet, this should be your internet (WAN)  IP.  Likewise,  if  you
               connect to a hub on your LAN, this should be your LAN IP.
 
-              Note that this setting is global for ncdc: it is currently  not  possible
-              to  use  a  single  instance  of ncdc to connect to both internet and LAN
-              hubs, if you are not reachable on the same IP with both networks. In that
-              case  you  can  either  use passive mode or run two separate instances of
-              ncdc.
+              Unlike  the  other  connection-related settings, this can be changed on a
+              per-hub basis, allowing you to have a different  public  IP  address  per
+              hub.
 
        active_port <integer>
               The  listen  port  for incoming connections in active mode. Set to `0' to
@@ -339,11 +344,12 @@ SETTINGS
               Set to true to automatically connect to the current hub when ncdc  starts
               up.
 
-       autorefresh <integer>
-              The time between automatic file refreshes, in minutes. Set to 0  to  dis‐
-              able automatically refreshing the file list. This setting also determines
-              whether ncdc will perform a refresh on startup. See the  `/refresh'  com‐
-              mand to manually refresh your file list.
+       autorefresh <interval>
+              The  time  between  automatic file refreshes. Recognized suffices are 's'
+              for seconds, 'm' for minutes, 'h' for hours and 'd' for days. Set to 0 to
+              disable  automatically refreshing the file list. This setting also deter‐
+              mines whether ncdc will perform a refresh on startup. See the  `/refresh'
+              command to manually refresh your file list.
 
        backlog <integer>
               When  opening  a  hub  or PM tab, ncdc can load a certain amount of lines
@@ -683,18 +689,17 @@ FILES
               should be generated automatically when ncdc starts up the first time, but
               can be generated manually using ncdc-gen-cert(1).
 
-       $NCDC_DIR/config.ini
-              Main  configuration  file.  It is recommended to use commands within ncdc
-              instead of editing this file manually. In particular, do  not  edit  this
-              file while ncdc is running, otherwise you will lose your changes.
+       $NCDC_DIR/db.sqlite3
+              The  database.  This  stores  all  configuration  variables, hash data of
+              shared files, download queue information  and  other  state  information.
+              Manually  editing this file with the `sqlite3' commandline tool is possi‐
+              ble but discouraged. Any changes made to the database while ncdc is  run‐
+              ning will not be read, and may even get overwritten by ncdc.
 
        $NCDC_DIR/dl/
               Directory  where  completed  downloads  are  moved  to by default. Can be
               changed with the `download_dir' configuration option.
 
-       $NCDC_DIR/dl.dat
-              Stores information about the download queue.
-
        $NCDC_DIR/files.xml.bz2
               Filelist containing a listing of all shared files.
 
@@ -704,9 +709,6 @@ FILES
               ncdc. Old file lists  are  deleted  automatically  after  a  configurable
               interval. See the `filelist_maxage' configuration option.
 
-       $NCDC_DIR/hashdata.dat
-              File information and hash data for shared files.
-
        $NCDC_DIR/history
               Command history.
 
@@ -750,8 +752,8 @@ AUTHOR
 
 
 SEE ALSO
-       ncdc-gen-cert(1).
+       ncdc-gen-cert(1), ncdc-db-upgrade(1).
 
 
 
-ncdc-1.5                             November 2011                              ncdc(1)
+ncdc-1.6                             December 2011                              ncdc(1)

download/ncdc-1.6.tar.gz.asc

@@ -0,0 +1,17 @@
+-----BEGIN PGP SIGNATURE-----
+Version: GnuPG v1.4.11 (GNU/Linux)
+
+iQIcBAABCgAGBQJO34UeAAoJEGI5TGmMJzn6wBkP/jZvIr0ylFoTwgSVEA9yaePk
+OcafdIaAuhJxk5b66cohmYSO5kkSWY9FtPtJYitdocLvzmLL5EkrOvY+WHCcBDcB
+kJa31ScqgfwXGIHbpaZU9aAwoU9uoAjzxoRalxCvQ3PHAp/l5i5NJMNxH2IpByRn
+Jr8Ufb6lj8Nkek9USl4NmYeK2LMQH/7q9FHr6gS82K1lJ0tNsYPYiBNF7eGjr1Bw
+pxoOthEtCbFeTIfJj2crM8UbQfXMsDFJHTbMmnh8dsAixIIpvT4CDR4Opaubp0Vc
+XdM+52W5EPPwGE8GdGc+r2X82LePB8zsjNmWoe7gGWH9nlMbH+IPDzsUj/oA+bj/
+L3T8/Uxs+uh/Jq3IX8v0WxMVcHVDIa8AE4yIRAs5apJwd9635V3CdDwGwIubHGXS
+GSSe7UjxZZ9sAIxwsmwgxamtW1+GjLEjzieuHQedVJNW2ObE8/pmpZfJBBDc0LqB
+wlpQsOzFH8R7y0HmB0VTnjerkbbaGKPOFVtYVFi04bjEc0J/XSXGWmHGT45Y0v7a
+Gs0ibEPjgTAZUQMjhj0U1qvv9cVmxJfsWwI9e48K1alCQy8rdwf+pF9PQYrzrlaB
+Sz90KQEuRM54H8VHWZg60Ba2YPv0S+xSqH8gbihdbxr6IEBLcUfkCmexzrudE5TV
+xMIJUXseGiv0A/C6K5pB
+=Z7Rl
+-----END PGP SIGNATURE-----

download/ncdc-1.6.tar.gz.md5

@@ -0,0 +1 @@
+99d5f58029c2676c38d417bf50632c3e  ncdc-1.6.tar.gz

download/ncdc-1.6.tar.gz.sha1

@@ -0,0 +1 @@
+e84831f08aa406134e14c5d14bce158d79b152dd  ncdc-1.6.tar.gz

index.cgi

@@ -56,6 +56,7 @@ sub home {
    E;
   end;
   h2 'Updates';
+  b '2011-12-07'; txt ' ncdc 1.6 released!'; br;
   b '2011-11-26'; txt ' Added article section and the article on SQLite.'; br;
   b '2011-11-03'; txt ' ncdc 1.5 and ncdu 1.8 released!'; br;
   b '2011-10-26'; txt ' ncdc 1.4 released!'; br;
@@ -251,13 +252,14 @@ sub ncdc {
 
   h2 'Get ncdc!';
   p;
-   b 'Latest version:'; txt ' 1.5 (';
-   $s->htmlDLLink('ncdc-1.5.tar.gz', 'download');
+   b 'Latest version:'; txt ' 1.6 (';
+   $s->htmlDLLink('ncdc-1.6.tar.gz', 'download');
    txt ' - '; a href => '/ncdc/changes', 'changes';
    txt ' - '; a href => 'https://sourceforge.net/projects/ncdc/files/ncdc/', 'mirror';
    txt ')'; br;
-   txt 'You can also get the latest development version from ';
-   a href => 'http://g.blicky.net/ncdc.git/', 'this git repository';
+   txt 'The latest development version can be fetched from the git repository
+        at '; code 'git://g.blicky.net/ncdc.git'; txt ' and is available for ';
+   a href => 'http://g.blicky.net/ncdc.git/', 'online browsing';
    txt '. The README includes instructions to build ncdc. Check out the ';
    a href => '/ncdc/man', 'manual'; txt ' to get started.';
    br;br;
@@ -311,7 +313,7 @@ sub ncdc {
   h2 'Requirements';
   p;
    lit <<'   E;';
-    The following libraries are required: ncurses, bzip2, gdbm, glib2 and libxml2.<br />
+    The following libraries are required: ncurses, bzip2, sqlite3, glib2 and libxml2.<br />
     For TLS support, you will need at least glib2 version 2.28.0 and
     glib-networking installed.<br />
     These dependencies should be easy to satisfy. Depending on your system, you
@@ -363,6 +365,8 @@ sub ncdcscr {
   my $s = shift;
   $s->htmlHeader(title => 'Screenshots', page => 'ncdc', tab => 'scr');
   p;
+   txt 'Note: While these screenshots are from version 1.5, the latest version has only little visible changes.';
+   br; br;
    txt 'Main chat';
    img src => '/img/ncdchub.png', class => 'scr', alt => 'Ncdc in the mainchat.';
    br;br;
@@ -497,6 +501,21 @@ sub docindex {
   p;
    txt '2011-11-26 - '; b 'Multi-threaded Access to an SQLite3 Database';
    txt ' ['; a href => '/doc/sqlaccess', 'HTML'; txt ' - '; a href => '/dat/sqlaccess', rel => 'nofollow', 'POD'; txt ']';
+   br;
+   txt '2010-06-02 - '; b 'Design and implementation of a compressed linked list library';
+   txt ' ['; a href => '/download/doc/compll.pdf', 'PDF'; txt ']';
+   br;br;
+   txt q|Disclaimer: The last one isn't really an article. It's the report for the
+       final project of my professional (HBO) bachelor of Electrical
+       Engineering. I was very liberal with some terminology in this report. For
+       example, "linked lists" aren't what you think they are, and I didn't even
+       use the term "locality of reference" where I really should have. It was
+       also written for an audience with little knowledge on the subject, so I
+       elaborated on a lot of things that should be obvious for most people in
+       the field. Then there is a lot of uninteresting overhead about the
+       project itself, which just happened to be mandatory for this report.
+       Nonetheless, if you can ignore these faults it's not such a bad read, if
+       I may say so myself. :-)|;
   end;
   $s->htmlFooter;
 }
@@ -590,6 +609,25 @@ sub dump {
    E;
    a href => 'http://g.blicky.net/serika.git/tree/json.mll', 'source';
   end;
+
+  h2 'ncdc-transfer-stats';
+  p;
+   lit <<'   E;';
+    September 2011. <a href="/ncdc">ncdc</a> gained transfer logging features,
+    and I wrote a quick Perl script to fetch some simple statistics from it.
+   E;
+   a href => 'http://p.blicky.net/agolr', 'source';
+  end;
+
+  h2 'ncdc-share-report';
+  p;
+   lit <<'   E;';
+    December 2011. Playing around with the Go programming language, I wrote
+    another transfer log parser and statistics generator for ncdc.
+    <a href="http://p.blicky.net/6yx2d">latest source</a>
+    (<a href="http://p.blicky.net/ab4lm">0.1</a>)
+   E;
+  end;
   $s->htmlFooter;
 }
 
@@ -823,9 +861,9 @@ sub dumpnccolour {
    li 'Unfortunately, not all terminals are configured in such a way that all
      possible colours are readable. So as a developer you\'ll still have to
      support configurable colour schemes in your ncurses application. :-(';
-   li 'None of the tested terminals make an attempt to change the foreground
-     colour if it is (almost) invisible on the selected background colour. This
-     can be a good thing or a bad thing, depending on what you want.';
+   li 'On most terminals, setting the foreground and background colour to the
+     same value without applying the A_BOLD attribute will make the text
+     invisible. Don\'t rely on this, however, as this is not the case on OS X.';
   end;
 
   h2 'Full screenshot';
@@ -844,8 +882,11 @@ sub dumpnccolour {
     'rox-t'        => 'Arch Linux, Roxterm, Tango color scheme',
     'rox-c'        => 'Arch Linux, Roxterm, Modified Tango color scheme',
     xterm          => 'Arch Linux, xterm (default settings)',
+    ubuntu         => 'Ubuntu 11.10, Gnome-terminal',
     debian         => 'Debian Squeeze, VT (default settings)',
     fbsd           => 'FreeBSD, VT (default settings)',
+    'osx-terminal' => 'Mac OS X, Terminal',
+    'osx-iterm2'   => 'Mac OS X, iTerm2',
   );
   while(@img) {
     my($n, $t) = (shift(@img), shift(@img));

robots.txt

@@ -1,3 +1,4 @@
 User-Agent: *
 Disallow: /download
+Disallow: /dat