vndb/yhdev
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Should have committed earlier

Browse source 
Lots of changes:
- Article about IPC
- New TUWF release
- New ncdu release
- Atom feeds for the bug tracker
- Bug tracker switch to sqlite
This commit is contained in:
Yorhel 2016-06-18 15:18:24 +02:00
parent 3b837d8564
commit 7cf772d968
33 changed files with 978 additions and 159 deletions
Download patch file Download diff file Expand all files Collapse all files

16
dat/doc
View file

@ -8,17 +8,21 @@ rare occasions are published on this page.
=over
=item C<2014-01-09 > - L<Some Measurements on Direct Connect File Lists|http://dev.yorhel.nl/doc/dcstats>
=item C<2014-07-29 > - L<The Sorry State of Convenient IPC|https://dev.yorhel.nl/doc/easyipc>
A long rant about IPC systems.
=item C<2014-01-09 > - L<Some Measurements on Direct Connect File Lists|https://dev.yorhel.nl/doc/dcstats>
A short measurement study on the file lists obtained from a Direct Connect hub.
Lots of graphs!
=item C<2012-02-15 > - L<A Distributed Communication System for Modular Applications|http://dev.yorhel.nl/doc/commvis>
=item C<2012-02-15 > - L<A Distributed Communication System for Modular Applications|https://dev.yorhel.nl/doc/commvis>
In this article I explain a vision of mine, and the results of a small research
project aimed at realizing that vision.
=item C<2011-11-26 > - L<Multi-threaded Access to an SQLite3 Database|http://dev.yorhel.nl/doc/sqlaccess>
=item C<2011-11-26 > - L<Multi-threaded Access to an SQLite3 Database|https://dev.yorhel.nl/doc/sqlaccess>
So you have a single database and some threads. How do you combine these in a
program?
@ -29,15 +33,15 @@ program?
=over
=item C<2014-06-10 > - L<Biased Random Periodic Switching in Direct Connect|http://dev.yorhel.nl/download/doc/brpsdc.pdf> (PDF)
=item C<2014-06-10 > - L<Biased Random Periodic Switching in Direct Connect|https://dev.yorhel.nl/download/doc/brpsdc.pdf> (PDF)
My masters thesis.
=item C<2013-04-05 > - L<Peer Selection in Direct Connect|http://dev.yorhel.nl/download/doc/psdc.pdf> (PDF)
=item C<2013-04-05 > - L<Peer Selection in Direct Connect|https://dev.yorhel.nl/download/doc/psdc.pdf> (PDF)
The rather long-ish literature study that precluded my masters thesis.
=item C<2010-06-02 > - L<Design and implementation of a compressed linked list library|http://dev.yorhel.nl/download/doc/compll.pdf> (PDF)
=item C<2010-06-02 > - L<Design and implementation of a compressed linked list library|https://dev.yorhel.nl/download/doc/compll.pdf> (PDF)
The report for the final project of my professional (HBO) bachelor of
Electrical Engineering. I was very liberal with some terminology in this

2
dat/doc-commvis
View file

@ -2,7 +2,7 @@ A Distributed Communication System for Modular Applications
=pod
(Published on B<2012-02-15>. Also available in L<POD|http://dev.yorhel.nl/dat/doc-commvis>.)
(Published on B<2012-02-15>. Also available in L<POD|https://dev.yorhel.nl/dat/doc-commvis>.)
=head1 Introduction

6
dat/doc-dcstats
View file

@ -7,8 +7,8 @@ Some Measurements on Direct Connect File Lists
=head1 Introduction
I've been working on Direct Connect related projects for a while now. This
includes maintaining L<ncdc|http://dev.yorhel.nl/ncdc> and
L<Globster|http://dev.yorhel.nl/globster>, and doing a bit of research into
includes maintaining L<ncdc|https://dev.yorhel.nl/ncdc> and
L<Globster|https://dev.yorhel.nl/globster>, and doing a bit of research into
improving the downloading performance and scalability (to be published at some
later date). Whether I'm writing code or trying to setup experiments for
research, there's one thing that helps a lot in making decisions. Measurements
@ -83,7 +83,7 @@ others will throw extra hardware at it, and I did what I do best: Optimize away
the constants. That is, I rewrote the data analysis program in C. Using the
excellent L<khash|https://github.com/attractivechaos/klib> hash table library
to keep track of the file information and the equally awesome
L<yxml|http://dev.yorhel.nl/yxml> library (a little bit of self-promotion
L<yxml|https://dev.yorhel.nl/yxml> library (a little bit of self-promotion
doesn't hurt, right?) to do the XML parsing, I was able to do all the necessary
processing in 30 minutes using at most 3.6GB of RAM.

683
dat/doc-easyipc Normal file
View file

@ -0,0 +1,683 @@
=pod
(Published on B<2014-07-29>.)
=head1 The Problem
How do you implement communication between two or more processes? This is a
question that has been haunting me for at least 6 years now. Of course, this
question is very broad and has many possible answers, depending on your
scenario. So let me get more specific by describing the problem I want to
solve.
What I want is to write a daemon process that runs in the background and can be
controlled from other programs or libraries. The intention is that people can
easily write custom interfaces or quick scripts to control the daemon. The
service that the daemon offers over this communication channel can be thought
of as its primary API, in this way you can think of the daemon as a persistent
programming library. This concept is similar to existing programs such as
L<btpd|https://github.com/btpd/btpd>, L<MPD|http://www.musicpd.org/>,
L<Transmission|https://www.transmissionbt.com/> and
L<Telepathy|http://telepathy.freedesktop.org/wiki/> - I'll get back to these
later.
More specifically, the most recent project I've been working on that follows
this pattern is L<Globster|https://dev.yorhel.nl/globster>, a remotely
controllable Direct Connect client (if you're not familiar with Direct Connect,
think of it as IRC with some additional file sharing capabilities built in).
While the problem I describe is not specific to Globster, it still serves as an
important use case. I see many other projects with similar IPC requirements.
The IPC mechanism should support two messaging patterns: Request/response and
asynchronous notifications. The request/response pattern is what you typically
get in RPC systems - the client requests something of the daemon and the daemon
then replies with a response. Asynchronous notifications are useful in allowing
the daemon to send asynchronous status updates to the client, such as incoming
chat messages or file transfer status. Lack of support for such notifications
would mean that a client needs to continuously poll for updates, which is
inefficient.
So what I'm looking for is a high-level IPC mechanism that handles this
communication. Solutions are evaluated by the following criteria, in no
particular order.
=over
=item B<Easy>
And with I<easy> I refer to I<ease of use>. As mentioned above, other people
should be able to write applications and scripts to control the daemon. Not
many people are willing to invest days of work just to figure out how to
communicate with the daemon.
=item B<Simple>
Simplicity refers to the actual protocol and the complexity of the code
necessary to implement it. Complex protocols require complex code, and complex
code is hard to maintain and will inevitably contain bugs. Note that I<simple>
and I<easy> are very different things and often even conflict with each other.
=item B<Small>
The IPC implementation shouldn't be too large, and shouldn't depend on huge
libraries. If you need several megabytes worth of libraries just to send a few
messages over a socket, you're doing it wrong.
=item B<Language independent>
Control the daemon with whatever programming language you're familiar with.
=item B<Networked>
A good solution should be accessible from both the local system (daemon running
on the same machine as the client) and from the network (daemon and client
running different machines).
=item B<Secure>
There's three parts in having a secure IPC mechanism. One part is to realize
that IPC operates at a I<trust boundary>; The daemon can't blindly trust
everything the client says and vice versa, so message validation and other
mechanisms to prevent DoS or information disclosure on either part are
necessary.
Then there the matter of I<confidentiality>. On a local system, UNIX sockets
will provide all the confidentiality you can get, so that's trivial. Networked
access, on the other hand, requires some form of transport layer security.
And finally, we need some form of I<authentication>. There should be some
mechanism to prevent just about anyone to connect to the daemon. A
coarse-grained solution such as file permissions on a local UNIX socket or a
password-based approach for networked access will do just fine for most
purposes. Really, just keep it simple.
=item B<Fast>
Although performance isn't really a primary goal, the communication between the
daemon and the clients shouldn't be too slow or heavyweight. For my purposes,
anything that supports about a hundred messages a second on average hardware
will do perfectly fine. And that shouldn't be particularly hard to achieve.
=item B<Proxy support>
This isn't really a hard requirement either, but it would be nice to allow
other processes (say, plugins of the daemon, or clients connecting to the
daemon) to export services over the same IPC channel as the main daemon. This
is especially useful in implementing a cross-language plugin architecture. But
again, not a hard requirement, because even if the IPC mechanism doesn't
directly support proxying, it's always possible for the daemon to implement
some custom APIs to achieve the same effect. This, however, requires extra work
and may not be as elegant as a built-in solution.
=back
Now let's discuss some existing solutions...
=head1 Custom Protocol
Why use an existing IPC mechanism in the first place when all you need is
UNIX/TCP sockets? This is the approach taken by
L<btpd|https://github.com/btpd/btpd>, L<MPD|http://www.musicpd.org/>
(L<protocol spec|http://www.musicpd.org/doc/protocol/index.html>) and older
versions of Transmission (see their L<1.2x
spec|https://trac.transmissionbt.com/browser/branches/1.2x/doc/ipcproto.txt>).
Brpd hasn't taken the time to documented the protocol format, suggesting it's
not really intended to be used as a convenient API (other than through their
btcli), and Transmission has since changed to a different protocol. I'll mainly
focus on MPD here.
MPD uses a text-based request/response mechanism, where each request is a
simple one-line command and a response consists of one or more lines, ending
with an C<OK> or C<ACK> line. There's no support for asynchronous
notifications, although that could obviously have been implemented, too. Let's
grade this protocol...
=over
=item B<Easy?> Not really.
Although MPD has conventions for how messages are formatted, each individual
message still requires custom parsing and validation. This can be automated by
designing an
L<IDL|https://en.wikipedia.org/wiki/Interface_description_language> and
accompanying code generator, but writing one specific for a single project
doesn't seem like a particularly fun task.
The protocol, despite its apparent simplicity, is apparently painful enough to
use that there is a special I<libmpdclient> library to abstract away the
communication with MPD, and interfaces to this library are available in many
programming languages. If you have access to such an application-specific
library for your language of choice, then sure, using the IPC mechanism is easy
enough. But that applies to literally any IPC mechanism.
Ideally, such a library needs to be written only once for the IPC mechanism in
use, and after that no additional code is needed to communicate with
services/daemons using that particular IPC mechanism. Code re-use among
different projects is great, yo. It also doesn't scale very well when extending
the services offered by daemon, any addition to the API will require
modifications to all implementations.
=item B<Simple?> Definitely.
I only needed a quick glance at the MPD protocol reference and I was able to
play a bit with telnet and control my MPD. Writing an implementation doesn't
seem like a complex task. Of course, this doesn't necessarily apply to all
custom protocols, but you can make it as simple or complex as you want it to
be.
=item B<Small?> Sure.
This obviously depends on how elaborate you design your protocol. If you have a
large or complex API, the size of a generic message parser and validator can
easily compensate for the custom parser and validator needed for each custom
message. But for a simple APIs, it's hard to beat a custom protocol in terms of
size.
=item B<Language independent?> Depends.
Of course, a socket library is available to most programming languages, and in
that sense any IPC mechanism built on sockets is language independent. This is,
as such, more of an argument as to how convenient it is to communicate with the
protocol directly rather than with a library that abstracts the protocol away.
In the case of MPD, the text-based protocol seems easy enough to use directly
from most languages, yet for some reason most people prefer language-specific
libraries for MPD.
If you design a binary protocol or anything more complex than simple
request/response message types, using your protocol directly is going to be a
pain in certain languages, and people will definitely want a library specific
to your daemon for their favourite programming language. Something you'll want
to avoid, I suppose.
=item B<Networked?> Sure enough.
Just a switch between UNIX sockets and TCP sockets. Whether a simple solution
like that is a good idea, however, depends on the next point...
=item B<Secure?> Ugh.
Security is hard to get right, so having an existing infrastructure that takes
care of most security sensitive features will help a lot. Implementing your own
protocol means that you also have to implement your own security, to some
extent at least.
Writing code to parse and validate custom messages is error-prone, and a bug in
this code could make both the daemon and the client vulnerable to crashes and
buffer overflows. A statically-typed abstraction that handles parsing and
validation would help a lot.
For networked communication, you'll need some form of confidentiality. MPD does
not seem to support this, so any networked access to an MPD server is
vulnerable to passive observers and MITM attacks. This may be fine for a local
network (presumably what it is intended to be used for), but certainly doesn't
work for exposing your MPD control interface to the wider internet. Existing
protocols such as TLS or SSH can be used to create a secure channel, but these
libraries tend to be large and hard to use securely. This is especially true
for TLS, but at least there's L<stunnel|https://www.stunnel.org/> to simplify
the implementation - at the cost of less convenient deployment.
In terms of authentication, you again need to implement this yourself. MPD
supports authentication using a plain-text password. This is fine for a trusted
network, but on an untrusted network you certainly want confidentiality to
prevent a random observer from reading your password.
=item B<Fast?> Sure.
Existing protocols may have put more effort into profiling and implementing
various optimizations than one would typically do with a custom and
quickly-hacked-together protocol, but still, it probably takes effort to design
a protocol that isn't fast enough.
=item B<Proxy support?> Depends...
Really depends on how elaborate you want to be. It can be very simple if all
you want is to route some messages, it can get very complex if you want to
ensure that these messages follow some format or if you want to reserve certain
interfaces or namespaces to certain clients. What surprised me about the MPD
protocol is that it actually has L<some support for
proxying|http://www.musicpd.org/doc/protocol/ch03s11.html>. But considering the
ad-hoc nature of the MPD protocol, the primitiveness and simplicity of this
proxy support wasn't too surprising. Gets the job done, I suppose.
=back
Overall, and as a rather obvious conclusion, a custom protocol really is what
you make of it. In general, though, it's a lot of work, not always easy to use,
and a challenge to get the security part right.
=head1 D-Bus
D-Bus is being used in L<Transmission|https://www.transmissionbt.com/> and is
what I used for L<Globster|https://dev.yorhel.nl/globster>.
On a quick glance, D-Bus looks I<perfect>. It is high-level, has the messaging
patterns I described, the L<protocol
specification|http://dbus.freedesktop.org/doc/dbus-specification.html> does not
seem I<overly> complex (though certainly could be simplified), it has
implementations for a number of programming languages, has support for
networking, proxying is part of normal operation, and it seems fast enough for
most purposes. When you actually give it a closer look, however, reality isn't
as rose-colored.
D-Bus is designed for two very specific use-cases. One is to allow local
applications to securely interact with system-level daemons such as
L<HAL|https://en.wikipedia.org/wiki/HAL_(software)> (now long dead) and
L<systemd|http://freedesktop.org/wiki/Software/systemd/>, and the other
use-case is to allow communication between different applications inside one
login session. As such, on a typical Linux system there are two D-Bus daemons
where applications can export interfaces and where messages can be routed
through. These are called the I<system bus> and the I<session bus>.
=over
=item B<Easy?> Almost.
The basic ideas behind D-Bus seem easy enough to use. The fact that is has
type-safe messages, interface descriptions and introspection really help in
making D-Bus a convenient IPC mechanism.
The main reasons why I think D-Bus isn't all that easy to use in practice is
due to the lack of good introductionary documentation and the crappy state of
the various D-Bus implementations. There is a L<fairly good
article|https://pythonhosted.org/txdbus/dbus_overview.html> providing a
high-level overview to D-Bus, but there isn't a lot of material that covers how
to actually use D-Bus to interact with applications or to implement a service.
On the implementations, I have had rather bad experiences with the actual
libraries. I've personally used the official libdbus-1, which markets itself a
"low-level" library designed to facilitate writing bindings for other
languages. In practice, the functionality that it offers appears to be too
high-level for writing bindings (L<GDBus|https://developer.gnome.org/glib/>
doesn't use it for this reason), and it is indeed missing a lot of
functionality to make it convenient to use directly. I've also played around
with Perl's L<Net::DBus|http://search.cpan.org/perldoc?Net%3A%3ADBus> and was
highly disappointed. Not only is the documentation rather incomplete, the
actual implementation has more bugs than features. And instead of building on
top of one of the many good event loops for Perl (such as
L<AnyEvent|http://search.cpan.org/perldoc?AnyEvent>), it chooses to implement
L<its own event
loop|http://search.cpan.org/perldoc?Net%3A%3ADBus%3A%3AReactor>. The existence
of several different libraries for Python doesn't incite much confidence,
either.
I was also disappointed in terms of the available tooling to help in the
development, testing and debugging of services. The L<gdbus(1)> tool is useful
for monitoring messages and scripting some things, but is not all that
convenient because D-Bus has too many namespaces and the terrible Java-like
naming conventions make typing everything out a rather painful experience.
L<D-Feet|http://live.gnome.org/DFeet/> offers a great way to explore services,
but lacks functionality for quick debugging sessions. I L<made an
attempt|http://g.blicky.net/dbush.git/> to write a convenient command-line
shell, but lost interest halfway. :-(
D-Bus has the potential to be an easy and convenient IPC mechanism, but the
lack of any centralized organization to offer good implementations,
documentation and tooling makes using D-Bus a pain to use.
=item B<Simple?> Not quite.
D-Bus is conceptually easy and the message protocol is alright, too. Some
aspects of D-Bus, however, are rather more complex than they need to be.
I have once made an attempt to fully understand how D-Bus discovers and
connects to the session bus, but I gave up halfway because there are too many
special cases. To quickly summarize what I found, there's the
C<DBUS_SESSION_BUS_ADDRESS> environment variable which could point to the
(filesystem or abstract) path of a UNIX socket or a TCP address. If that
variable isn't set, D-Bus will try to connect to your X server and get the
address from that. In order to avoid linking everything against X libraries, a
separate L<dbus-launch> utility is spawned instead. Then the bus address could
also be obtained from a file in your C<$HOME/.dbus/> directory, with added
complexity to still support a different session bus for each X session. I've no
idea how exactly connection initiation to the system bus works, but my
impression is that a bunch of special cases exist there, too, depending on
which init system your OS happens to use.
As if all the options in connection initiation aren't annoying enough, there's
also work on L<kdbus|https://lwn.net/Articles/580194/>, a Linux kernel
implementation to get better performance. Not only will kdbus use a different
underlying communication mechanism, it will also switch to a completely
different serialization format. If/when this becomes widespread you will have
to implement and support two completely different protocols and pray that your
application works with both.
On the design aspect there is, in my opinion, needless complexity with regards
to naming and namespaces. First there is a global namespace for I<bus names>,
which are probably better called I<application names>, because that's usually
what they represent. Then, there is a separate I<object> namespace local to
each bus name. Each object has methods and properties, and these are
associated with an I<interface name>, in a namespace specific to the particular
object. Despite these different namespaces, the convention is to use a full and
globally unique path for everything that has a name. For example, to list the
IM protocols that Telepathy supports, you call the C<ListProtocols> method in
the C<org.freedesktop.Telepathy.ConnectionManager> interface on the
C</org/freedesktop/Telepathy/ConnectionManager> object at the
C<org.freedesktop.Telepathy> bus. Fun times indeed. I can understand the
reasoning behind most of these choices, but in my opinion they found the wrong
trade-off.
Another point of complexity that annoys me is the fact that an XML format is
used to describe interfaces. Supporting XML as an IDL format is alright, but
requiring a separate format for an introspection interface gives me the
impression that the message format wasn't powerful enough for such a simple
purpose. The direct effect of this is that any application wishing to use
introspection data will have to link against an XML parser, and almost all
conforming XML parser implementations are as large as the D-Bus implementation
itself.
=item B<Small?> Kind of.
C<libdbus-1.so.3.8.6> on my system is about 240 KiB. It doesn't cover parsing
interface descriptions or implementing a D-Bus daemon, but still covers most of
what is needed to interact with services and to offer services over D-Bus.
It's not I<that> small, but then again, libdbus-1 was not really written with
small size in mind. There's room for optimization.
=item B<Language independent?> Sure.
D-Bus libraries exist for a number of programming languages.
=item B<Networked?> Half-assed.
D-Bus I<officially> supports networked connections to a D-Bus daemon. Actually
using this, however, is painful. Convincing L<dbus-daemon(1)> to accept
connections on a TCP socket involves disabling all authentication (it expects
UNIX credential passing, normally) and requires adding an undocumented C<<
<allow_anonymous/> >> tag in the configuration (I only figured this out from
reading the source code).
Even when you've gotten that to work, there is the problem that D-Bus isn't
totally agnostic to the underlying socket protocol. D-Bus has support for
passing UNIX file descriptors over the connection, and this of course doesn't
work over TCP. While this feature is optional and easily avoided, some services
(I can't find one now) use UNIX fds in order to keep track of processes that
listen to a certain event. Obviously, those services can't be accessed over the
network.
=item B<Secure?> Only locally.
D-Bus has statically typed messages that can be validated automatically, so
that's a plus.
For local authentication, there is support for standard UNIX permissions and
credential passing for more fine-grained authorization. For remote
authentication, I think there is support for a shared secret cookie, but I
haven't tried to use this yet.
There is, as with MPD, no support at all for confidentiality, so using
networked D-Bus over an untrusted network would be a very bad idea anyway.
=item B<Fast?> Mostly.
The messaging protocol is fairly lightweight, so no problems there. I do have
to mention two potential performance issues, however.
The first issue is that the normal mode of operation in D-Bus is to proxy all
messages through an intermediate D-Bus daemon. This involves extra context
switches and message parsing passes in order to get one message from
application A to application B. I believe it is I<officially> supported to
bypass this daemon and to communicate directly between two processes, but after
my experience with networking I am wary of trying anything that isn't part of
how D-Bus is I<intended> to be used. This particular performance issue is what
kdbus addresses, so I suppose it won't apply to future Linux systems.
The other issue is that a daemon that provides a service over D-Bus does not
know whether there exists an application that is interested receiving its
notifications. This means that the daemon always has to spend resources to send
out notification messages, even if no application is actually interested in
receiving them. In practice this means that the notification mechanism is
avoided for events that may occur fairly often, and an equally inefficient
polling approach has to be used instead. It is possible for a service provider
to keep track of interested applications, but this is not part of the D-Bus
protocol and not something you would want to implement for each possible event.
I've no idea if kdbus addresses this issue, but it would be stupid not to.
=item B<Proxy support?> Yup.
It's part of normal operation, even.
=back
D-Bus has many faults, some of them are by design, but many are fixable. I
would have contributed to improving the situation, but I get the feeling that
the goals of the D-Bus maintainers are not at all aligned with mine. My
impression is that the D-Bus maintainers are far too focussed on their own
specific needs and care little about projects with slightly different needs.
Especially with the introduction of kdbus, I consider D-Bus too complex now to
consider it worth the effort to improve. Starting from scratch seems less work.
=head1 JSON/XML RPC
While I haven't extensively used JSON-RPC or XML-RPC myself, it's still an
interesting alternative to study.
L<Transmission|https://www.transmissionbt.com/> uses JSON-RPC
(L<spec|https://trac.transmissionbt.com/browser/trunk/extras/rpc-spec.txt>) as
its primary IPC mechanism, and L<RTorrent|http://rakshasa.github.io/rtorrent/>
has support for an optional XML-RPC interface. (Why do I keep referencing
torrent clients? Surely there are other interesting applications? Oh well.)
The main selling point of HTTP-based IPC is that it is accessible from
browser-based applications, assuming everything has been setup correctly. This
is a nice advantage, but lack of this support is not really a deal-breaker for
me. Browser-based applications can still use any other IPC mechanism, as long
as there are browser plugins or some form of proxy server that converts the
messages of the IPC mechanism to something that is usable over HTTP. For
example, both solutions exist for D-Bus, in the form of the L<Browser DBus
Bridge|http://sandbox.movial.com/wiki/index.php/Browser_DBus_Bridge> and
L<cloudeebus|https://github.com/01org/cloudeebus>. Of course, such solutions
typically aren't as convenient as native HTTP support.
Since HTTP is, by design, purely request-response, JSON-RPC and XML-RPC don't
generally support asynchronous notifications. It's possible to still get
asynchronous notifications by using
L<WebSockets|https://en.wikipedia.org/wiki/WebSocket> (Ugh, opaque stream
sockets, time to go back to our L<custom protocol|/Custom Protocol>) or by
having the client implement a HTTP server itself and send its URL to the
service provider (This is known as a
L<callback|https://duckduckgo.com/?q=web%20service%20callback> in the
L<SOAP|https://en.wikipedia.org/wiki/SOAP> world. I have a lot of respect for
developers who can put up with that crap). As I already hinted, neither
solution is simple or easy.
Let's move on to the usual grading...
=over
=item B<Easy?> Sure.
The ubiquity of HTTP, JSON and XML on the internet means that most developers
are already familiar with using it. And even if you aren't, there are so many
easy-to-use and well-documented libraries available that you're ready to go in
a matter of minutes.
Although interface description languages/formats exist for XML-RPC (and
possibly for JSON-RPC, too), I get the impression these are not often used
outside of the SOAP world. As a result, interacting with such a service tends
be weakly/stringly typed, which, I imagine, is not as convenient in strongly
typed programming languages.
=item B<Simple?> Not really.
Many people have the impression that HTTP is somehow a simple protocol. Sure,
it may look simple on the wire, but in reality it is a hugely bloated and
complex protocol. I strongly encourage everyone to read through L<RFC
2616|https://tools.ietf.org/html/rfc2616> at least once to get an idea of its
size and complexity. To make things worse, there's a lot of recent activity to
standardize on a next generation HTTP
(L<SPDY|https://en.wikipedia.org/wiki/SPDY> and L<HTTP
2.0|https://en.wikipedia.org/wiki/HTTP_2.0>), but I suppose we can ignore these
developments for the foreseeable future for the use case of IPC.
Of course, a lot of the functionality specified for HTTP is optional and can be
ignored for the purpose of IPC, but that doesn't mean that these options don't
exist. When implementing a client, it would be useful to know exactly which
HTTP options the server supports. It would be wasteful to implement compression
support if the server doesn't support it, or keep-alive, or content
negotiation, or ranged requests, or authentication, or correct handling for all
response codes when the server will only ever send 'OK'. What also commonly
happens is that server implementors want to support as much as possible, to the
point that you can have JSON or XML output, depending on what the client
requested.
XML faces a similar problem. The format looks simple, but the specification has
a bunch of features that hardly anyone uses. In contrast to HTTP, however, a
correct XML parser can't just decide to not parse C<< <!DOCTYPE ..> >> stuff,
so it I<has> to implement some of this complexity.
On the upside, JSON is a really simple serialization format, and if you're
careful enough to only implement the functionality necessary for basic HTTP, a
JSON-RPC implementation I<can> be somewhat simple.
=item B<Small?> Not really.
What typically happens is that implementors take an existing HTTP library and
build on top of that. A generic HTTP library likely implements a lot more than
necessary for IPC, so that's not going to be very small. RTorrent, for example,
makes use of the not-very-small L<xmlrpc-c|http://xmlrpc-c.sourceforge.net/>,
which in turn uses L<libcurl|http://curl.haxx.se/> (400 KiB, excluding TLS
library) and either the bloated L<libxml2|http://xmlsoft.org/> (1.5 MiB) or
L<libexpat|http://www.libexpat.org/> (170 KiB). In any case, expect your
programs to grow by a megabyte or more if you go this route.
Transmission seems rather less bloated. It uses the HTTP library that is built
into L<libevent|http://libevent.org/> (totalling ~500 KiB, but libevent is also
used for other networking parts), and a simple JSON parser can't be that large
either. I'm sure that if you reimplement everything from scratch for the
purpose of building an API, you could get something much smaller. Then again,
even if you manage to shrink the size of the server that way, you can't expect
all your users to do the same.
If HTTPS is to be supported, add ~500 KiB more. TLS isn't the simplest
protocol, either.
=item B<Language independent?> Yes.
Almost every language has libraries for web stuff.
=item B<Networked?> Definitely.
In fact, I've never seen anyone use XML/JSON RPC over UNIX sockets.
=item B<Secure?> Alright.
HTTP has built-in support for authentication, but it also isn't uncommon to use
some other mechanism (based on cookies, I guess?).
Confidentiality can be achieved with HTTPS. There is the problem of verifying
the certificate, since I doubt anyone is going to have certificates of their
local applications signed by a certificate authority, but there's always the
option of trust-on-first use. Custom applications can also include a
fingerprint of the server certificate in the URL for verification, but this
won't work for web apps.
=item B<Fast?> No.
JSON/XML RPC messages add significant overhead to the network and requires more
parsing than a simple custom solution or D-Bus. I wouldn't really call it
I<fast>, but admittedly, it might still be I<fast enough> for most purposes.
=item B<Proxy support?> Sure.
HTTP has native support for proxying, and it's always possible to proxy some
URI on the main server to another server, assuming the libraries you use
support that. It's not necessarily simple to implement, however.
=back
The lack of asynchronous notifications and the overhead and complexity of
JSON/XML RPC make me stay away from it, but it certainly is a solution that
many client developers will like because of its ease of use.
=head1 Other Systems
There are a more alternatives out there than I have described so far. Most of
those were options I dismissed early on because they're either incomplete
solutions or specific to a single framework or language. I'll still mention a
few here.
=head2 Message Queues
In the context of IPC I see that message queues such as
L<RabbitMQ|https://www.rabbitmq.com/> and L<ZeroMQ|http://zeromq.org/> are
quite popular. I can't say I have much experience with any of these, but these
MQs don't seem to offer a solution to the problem I described in the
introduction. My impression of MQs is that they offer a higher-level and more
powerful alternative to TCP and UDP. That is, they route messages from one
endpoint to another. The contents of the messages are still completely up to
the application, so you're still on your own in implementing an RPC mechanism
on top of that. And for the purpose of building a simple RPC mechanism, I'm
convinced that plain old UNIX sockets or TCP will do just fine.
=head2 Cap'n Proto
I probably should be spending a full chapter on L<Cap'n
Proto|http://kentonv.github.io/capnproto/> instead of this tiny little section,
but I'm simply not familiar enough with it to offer any deep insights. I can
still offer my blatantly uninformed impression of it: It looks very promising,
but puts, in my opinion, too much emphasis on performance and too little
emphasis on ease of use. It lacks introspection and requires that clients have
already obtained the schema of the service in order to interact with it. It
also uses a capability system to handle authorization, which, despite being
elegant and powerful, increases complexity and cognitive load (though I
obviously need more experience to quantify this). It still lacks
confidentiality for networked access and the number of bindings to other
programming languages is limited, but these problems can be addressed.
Cap'n Proto seems like the ideal IPC mechanism for internal communication
within a single (distributed) application and offers a bunch of unique features
not found in other RPC systems. But it doesn't feel quite right as an easy API
for others to use.
=head2 CORBA
CORBA has been used by the GNOME project in the past, and was later abandoned
in favour of D-Bus, primarily (I think) because CORBA was deemed too L<complex
and incomplete|http://dbus.freedesktop.org/doc/dbus-faq.html#corba>. A system
that is deemed more complex than D-Bus is an immediate red flag. The L<long and
painful history of CORBA|http://queue.acm.org/detail.cfm?id=1142044> also makes
me want to avoid it, if only because that makes it very hard to judge the
quality and modernness of existing implementations.
=head2 Project Tanja
A bit over two years ago I was researching the same problem, but from a much
more generic angle. The result of that was a project that I called Tanja. I
described its concepts L<in an earlier
article|https://dev.yorhel.nl/doc/commvis>, and wrote an incomplete
L<specification|http://g.blicky.net/tanja.git/> along with implementations in
L<C|http://g.blicky.net/tanja-c.git/>, L<Go|http://g.blicky.net/tanja-go.git/>
and L<Perl|http://g.blicky.net/tanja-perl.git/>. I consider project Tanja a
failure, primarily because of its genericity. It supported too many
communication models and the lack of a specification as to which model was
used, and the lack of any guarantee that this model was actually followed, made
Tanja hard to use in practice. It was a very interesting experiment, but not
something I would actually use. I learned the hard way that you sometimes have
to move some complexity down into a lower abstraction layer in order to keep
the complexity in check at higher layers of abstraction.
=head1 Conclusions
This must be the longest rant I've written so far.
In any case, there isn't really a perfect IPC mechanism for my use case. A
custom protocol involves reimplementing a lot of stuff, D-Bus is a pain, and
JSON/XML RPC are bloat.
I am still undecided on what to do. I have a lot of ideas as to what a perfect
IPC solution would look like, both in terms of features and in how to implement
it, and I feel like I have enough experience by now to actually develop a
proper solution. Unfortunately, writing a complete IPC system with the required
utilities and language bindings takes B<a lot> of time and effort. It's not
really worth it if I am the only one using it.
So here is my plea to you, dear reader: If you know of any existing solutions
I've missed, please tell me. If you empathize with me and want a better
solution to this problem, please get in touch as well! I'd love to hear about
projects which face similar problems and have similar requirements.

20
dat/dump
View file

@ -5,7 +5,7 @@ for the purpose of getting something done. This page is a listing of those I
thought might be of useful to others as well.
I also maintain a collection of miscellaneous C micro-libraries. Those are
listed under the collective name of L<Ylib|http://dev.yorhel.nl/ylib>.
listed under the collective name of L<Ylib|https://dev.yorhel.nl/ylib>.
=head2 maildir.pl
@ -25,9 +25,9 @@ Download: L<0.3|http://p.blicky.net/h25z8>
=head2 ncdc-transfer-stats
September 2011. L<ncdc|http://dev.yorhel.nl/ncdc> gained transfer logging
September 2011. L<ncdc|https://dev.yorhel.nl/ncdc> gained transfer logging
features, and I wrote a quick Perl script to fetch some simple statistics from
it. L<source|http://p.blicky.net/agolr>
it. L<source|http://p.blicky.net/eu00a> (L<0.1|http://p.blicky.net/agolr>).
=head2 json.mll
@ -43,7 +43,7 @@ November 2009. The L<public VNDB API|http://vndb.org/d11> was designed to be
easy to use even from low level languages. I wrote this simple program to see
how much work it would be to use the API in C, and as example code for anyone
wishing to use the API for something more useful. Read the comments for more
info. L<source|http://dev.yorhel.nl/download/code/vinfo.c>
info. L<source|https://dev.yorhel.nl/download/code/vinfo.c>
=head2 Microdc2 log file parser
@ -51,25 +51,25 @@ June 2007. Simple perl script that parses log files created by
L<microdc2|http://corsair626.no-ip.org/microdc/> and outputs a simple and
ugly html file with all uploaded files. It correctly merges chunked
uploads, calculates average upload speed per file and total bandwidth used
for uploads. L<source|http://dev.yorhel.nl/download/code/mdc2-parse.pl>
for uploads. L<source|https://dev.yorhel.nl/download/code/mdc2-parse.pl>
B<Note:> for those of you who still use microdc2, please have a look at
L<ncdc|http://dev.yorhel.nl/ncdc>, a modern alternative.
L<ncdc|https://dev.yorhel.nl/ncdc>, a modern alternative.
=head2 yapong.c
Feburary 2006. Yet Another Pong, and yet another program written just for
testing/ learning purposes. Tested to work with the ncurses or pdcurses
libraries. L<source|http://dev.yorhel.nl/download/code/yapong.c> (L<older
version|http://dev.yorhel.nl/download/code/yapong-0.01.c>).
libraries. L<source|https://dev.yorhel.nl/download/code/yapong.c> (L<older
version|https://dev.yorhel.nl/download/code/yapong-0.01.c>).
=head2 echoserv.c
February 2006. A simple non-blocking single-threaded TCP echo server,
displaying how the select() system call can be used to handle multiple
connections. L<source|http://dev.yorhel.nl/download/code/echoserv.c>
connections. L<source|https://dev.yorhel.nl/download/code/echoserv.c>
=head2 bbcode.c
January 2006. Simple BBCode to HTML converter written in plain C, for learning
puroses. L<source|http://dev.yorhel.nl/download/code/bbcode.c>
puroses. L<source|https://dev.yorhel.nl/download/code/bbcode.c>

2
dat/dump-awshrink
View file

@ -6,7 +6,7 @@ page generation times for the AWStats pages. I wrote a small script that
analyzes these data files and can remove any information you think is
unnecessary.
B<Download:> L<awshrink|http://dev.yorhel.nl/download/code/awshrink> (copy to
B<Download:> L<awshrink|https://dev.yorhel.nl/download/code/awshrink> (copy to
/usr/bin to install).

2
dat/dump-demo
View file

@ -22,7 +22,7 @@ only 3 actual entries, but oh well), best non-windows 64k intro (it was the
only one in the competition), and the Digitale Kultur newcomer award, which
actually is something to be proud of, I guess.
L<download|http://dev.yorhel.nl/download/yorhel~bluecubes.zip> -
L<download|https://dev.yorhel.nl/download/yorhel~bluecubes.zip> -
L<mirror|http://scene.org/file.php?file=/parties/2006/evoke06/in64/yorhel_bluecubes.zip&fileinfo>
(includes linux binaries, windows port, and sources) -
L<pouet comments|http://pouet.net/prod.php?which=25866>.

2
dat/dump-grenamr
View file

@ -14,7 +14,7 @@ renaming of the file. There's one other variable that the rename command does
not have: $i, which reflects the file number (starting from 0) in the current
list. This allows expressions such as as C<$_=sprintf'%03d.txt',$i>.
B<Download: > L<grenamr|http://dev.yorhel.nl/download/code/grenamr-0.1.pl>
B<Download: > L<grenamr|https://dev.yorhel.nl/download/code/grenamr-0.1.pl>
(copy to /usr/bin/ to install)
Requires the Gtk2 Perl module. Most distributions have a perl-gtk2 package.

6
dat/dump-insbench
View file

@ -20,7 +20,7 @@ precise). My goal wasn't to benchmark the performance of different
implementations, so I simply chose two implementations that I suspect are among
the fastest. The vector implementation in the benchmarks is my own creation:
L<vec.h|http://g.blicky.net/globster.git/tree/src/util/vec.h?id=2c11d2a> from
the L<Globster|http://dev.yorhel.nl/globster> code base.
the L<Globster|https://dev.yorhel.nl/globster> code base.
B<Source code:> L<ins-bench.c|http://p.blicky.net/r746e>
@ -39,7 +39,7 @@ search. Actual performance will be thus be a bit worse, depending on whether
the final application needs that binary search or whether it can assume its
input to be already sorted.
L<[img graph insbench-bench-thumb.png ]|http://dev.yorhel.nl/img/insbench-bench.png>
L<[img graph insbench-bench-thumb.png ]|https://dev.yorhel.nl/img/insbench-bench.png>
Gnuplot script: (The awk(ward) part can likely be done natively in gnuplot as
well, but I was too lazy to figure out how)
@ -63,7 +63,7 @@ be a more accurate simulation of some real-world applications. This time I'm
not cheating with the vector implementation, a binary search is performed in
order to insert the items in the correct location.
L<[img graph insbench-rand-thumb.png ]|http://dev.yorhel.nl/img/insbench-rand.png>
L<[img graph insbench-rand-thumb.png ]|https://dev.yorhel.nl/img/insbench-rand.png>
set terminal png size 1000, 1500
set output "bench-rand.png"

4
dat/dump-nccolour
View file

@ -8,7 +8,7 @@ different terminals and different configurations. Note that only the 8 basic
NCurses colours are tested, the more flexible init_color() function is not
used.
B<Source code: > L<nccolour.c|http://dev.yorhel.nl/download/code/nccolour.c>
B<Source code: > L<nccolour.c|https://dev.yorhel.nl/download/code/nccolour.c>
(L<syntax highlighed version|http://p.blicky.net/xu35c>)
=head2 Notes / observations
@ -97,3 +97,5 @@ program, which also explains what each column means.
=item CentOS 6.4
[img scr nccol-centos64.png ]
=back

17
dat/globster
View file

@ -1,6 +1,15 @@
=pod
[html]<!-- This code is ugly as hell. -->
[html]<p><b style="color: #f00">Project Abandoned</b><br />
I've stopped development on Globster. I still believe the overall idea and
architecture of Globster are good, and the DC community would definitely
benefit from a remotely controllable client, but Globster in its current form
wasn't going into the direction I wanted it to. I might restart the project
from scratch (yet again) in the future, but for now... it's as dead as a cute
zombie whale.<br /><br />
</p>
<!-- This code is ugly as hell. -->
<div style="width: 600px; height: 227px; background-image: url(/img/globster.png); margin-bottom: -30px">
<b style="font-size: 14px; position: relative; left: 150px; top: 10px">The Globster What?</b>
<p style="position: relative; left: 150px; top: 20px; width: 420px; text-align: right">
@ -77,13 +86,13 @@ Globster isn't particularly hard to use, but usage documentation is currently a
bit lacking. I have every intention to fix that, but for now, you're encouraged
to join the development hub and bug me for help: C<adc://dc.blicky.net:2780/>.
I did already write some
L<API documentation|http://dev.yorhel.nl/globster/api>.
L<API documentation|https://dev.yorhel.nl/globster/api>.
There are at this point not many scripts or interfaces available for Globster:
=over
=item * L<globsterctl|http://dev.yorhel.nl/globster/ctl> - A control script for the daemon, included in the git repo.
=item * L<globsterctl|https://dev.yorhel.nl/globster/ctl> - A control script for the daemon, included in the git repo.
=item * L<globster-feedspam.pl|http://p.blicky.net/0z9uw> - An RSS / Atom notification script.
@ -107,7 +116,7 @@ Globster incorporates code from
L<libev|http://software.schmorp.de/pkg/libev.html>,
L<freetiger|http://klondike.es/freetiger/>,
L<klib|https://github.com/attractivechaos/klib> and
L<ylib|http://dev.yorhel.nl/ylib>.
L<ylib|https://dev.yorhel.nl/ylib>.
Additionally, L<autoconf-lean|https://bitbucket.org/GregorR/autoconf-lean> is
used to keep the configure script fast.

2
dat/globster-api
View file

@ -1 +1 @@
/home/yhdev/globster/doc/api.pod
../../globster/doc/api.pod

15
dat/ncdc
View file

@ -11,14 +11,13 @@ ncurses interface.
=item Latest version
1.19.1 ([dllink ncdc-1.19.1.tar.gz download]
- L<changes|http://dev.yorhel.nl/ncdc/changes>
- L<mirror|https://sourceforge.net/projects/ncdc/files/ncdc/>)
- L<changes|https://dev.yorhel.nl/ncdc/changes>)
Convenient static binaries for Linux:
L<64-bit|http://dev.yorhel.nl/download/ncdc-linux-x86_64-1.19.1-3-g8e3a7.tar.gz> -
L<32-bit|http://dev.yorhel.nl/download/ncdc-linux-i486-1.19.1-3-g8e3a7.tar.gz> -
L<ARM|http://dev.yorhel.nl/download/ncdc-linux-arm-1.19.1-3-g8e3a7.tar.gz>. Check the
L<installation instructions|http://dev.yorhel.nl/ncdc/install> for more info.
L<64-bit|https://dev.yorhel.nl/download/ncdc-linux-x86_64-1.19.1-12-g55616.tar.gz> -
L<32-bit|https://dev.yorhel.nl/download/ncdc-linux-i486-1.19.1-12-g55616.tar.gz> -
L<ARM|https://dev.yorhel.nl/download/ncdc-linux-arm-1.19.1-12-g55616.tar.gz>. Check the
L<installation instructions|https://dev.yorhel.nl/ncdc/install> for more info.
=item Development version
@ -36,9 +35,9 @@ Ncdc is entirely written in C and available under a liberal MIT license.
=item Community
[html]
L<Bug tracker|http://dev.yorhel.nl/ncdc/bug> - For bugs reports, feature requests and patches.<br />
L<Bug tracker|https://dev.yorhel.nl/ncdc/bug> - For bugs reports, feature requests and patches.<br />
L<Mailing list|http://l.blicky.net/listinfo/ncdc> - For general discussions, questions and feedback.<br />
C<adc://dc.blicky.net:2780/> - If the mailing list is too slow for you.
C<adcs://dc.blicky.net:2780/> - If the mailing list is too slow for you.
É
=item Packages and ports

12
dat/ncdc-faq
View file

@ -34,13 +34,19 @@ reimplementing everything: I get to choose the library dependencies and the
memory/CPU efficiency trade-offs, and I am not limited by an existing
implementation that needs quite a few modifications to achieve what I want.
Most of the "special features not commonly found in other clients" mentioned on
the L<homepage|http://dev.yorhel.nl/ncdc> are a direct result of this.
the L<homepage|https://dev.yorhel.nl/ncdc> are a direct result of this.
B<Personal preferences:> These are simple: I rather dislike C++ and working
with other people's code. Working with other people's C++ code isn't exactly
something I wish to spend my free time on.
=head2 Does ncdc support TLS 1.2?
Yes, but you need a recent version of GnuTLS. Nobody knows what counts as
"recent", exactly, but I'm guessing any 3.0+ version will do.
=head2 What protocol features does ncdc support?
For ADC: BASE, RF, TIGR, BZIP, BLOM, ADCS, KEYP and SUDP.
@ -96,7 +102,7 @@ a crash, please report a bug.
=head2 Ncdc uses too much disk space!
First, look where this disk space goes to (hint: use
L<ncdu|http://dev.yorhel.nl/ncdu>). If it's the log files: you can safely
L<ncdu|https://dev.yorhel.nl/ncdu>). If it's the log files: you can safely
delete or rotate them (see next question).
The I<db.sqlite3> file can also grow quite large in certain situations. If you
@ -141,7 +147,7 @@ myself, though. I just run ncdc directly on my router. :-)
=head2 Are there any programs available for analyzing the transfers.log file?
Nothing like that is included in the release yet, but there is a simple Perl
script available: L<ncdc-transfer-stats|http://p.blicky.net/agolr>, and a short
script available: L<ncdc-transfer-stats|http://p.blicky.net/eu00a>, and a short
Go program: L<ncdc-share-report|http://p.blicky.net/h25z8>.

14
dat/ncdc-install
View file

@ -9,7 +9,7 @@ In theory, the following instructions should work everywhere:
=item * Install the required dependencies: ncurses, bzip2, zlib, sqlite3, glib2 and gnutls,
=item * Download and extract the source tarball from the L<homepage|http://dev.yorhel.nl/ncdc>,
=item * Download and extract the source tarball from the L<homepage|https://dev.yorhel.nl/ncdc>,
=item * C<./configure>
@ -38,11 +38,11 @@ compiling and/or installing it, I also offer statically linked binaries:
=over
=item * L<Linux, 64-bit|http://dev.yorhel.nl/download/ncdc-linux-x86_64-1.19.1-3-g8e3a7.tar.gz>
=item * L<Linux, 64-bit|https://dev.yorhel.nl/download/ncdc-linux-x86_64-1.19.1-12-g55616.tar.gz>
=item * L<Linux, 32-bit|http://dev.yorhel.nl/download/ncdc-linux-i486-1.19.1-3-g8e3a7.tar.gz>
=item * L<Linux, 32-bit|https://dev.yorhel.nl/download/ncdc-linux-i486-1.19.1-12-g55616.tar.gz>
=item * L<Linux, ARM|http://dev.yorhel.nl/download/ncdc-linux-arm-1.19.1-3-g8e3a7.tar.gz>
=item * L<Linux, ARM|https://dev.yorhel.nl/download/ncdc-linux-arm-1.19.1-12-g55616.tar.gz>
=back
@ -130,7 +130,7 @@ First install some required packages (as root):
Then, fetch the ncdc source tarball, extract and build as follows:
wget http://dev.yorhel.nl/download/ncdc-1.19.1.tar.gz
wget https://dev.yorhel.nl/download/ncdc-1.19.1.tar.gz
tar -xf ncdc-1.19.1.tar.gz
cd ncdc-1.19.1
export PATH="$PATH:/usr/perl5/5.10.0/bin"
@ -180,7 +180,7 @@ required libraries:
Then run the following commands to download and install ncdc:
wget http://dev.yorhel.nl/download/ncdc-1.19.1.tar.gz
wget https://dev.yorhel.nl/download/ncdc-1.19.1.tar.gz
tar -xf ncdc-1.19.1.tar.gz
cd ncdc-1.19.1
./configure --prefix=/usr
@ -224,7 +224,7 @@ website|http://cygwin.com/> and use it to install the following packages:
Then open a Cygwin terminal and run the following commands to download,
compile, and install ncdc:
wget http://dev.yorhel.nl/download/ncdc-1.19.1.tar.gz
wget https://dev.yorhel.nl/download/ncdc-1.19.1.tar.gz
tar -xf ncdc-1.19.1.tar.gz
cd ncdc-1.19.1
./configure --prefix=/usr

4
dat/ncdc-man
View file

@ -907,7 +907,7 @@ more information.
=head1 BUGS
Please report bugs or feature requests to the bug tracker or the mailing list.
Both can be found on the ncdc homepage at L<http://dev.yorhel.nl/ncdc>. There
Both can be found on the ncdc homepage at L<https://dev.yorhel.nl/ncdc>. There
is also an ADC hub available at C<adc://dc.blicky.net:2780/> for general
support and discussions.
@ -915,4 +915,4 @@ support and discussions.
ncdc is written by Yoran Heling <projects@yorhel.nl>
Web: L<http://dev.yorhel.nl/ncdc>
Web: L<https://dev.yorhel.nl/ncdc>

36
dat/ncdu
View file

@ -1,11 +1,10 @@
=pod
Not quite happy with the available disk usage analyzers and looking for a fun
project to get used to C programming, I started working on ncdu: A disk usage
analyzer with an ncurses interface, aimed to be run on a remote server where
you don't have an entire gaphical setup, but have to do with a simple SSH
connection. ncdu aims to be fast, simple and easy to use, and should be able
to run in any minimal POSIX-like environment with ncurses installed.
Ncdu is a disk usage analyzer with an ncurses interface. It is designed to find
space hogs on a remote server where you don't have an entire graphical setup
available, but it is a useful tool even on regular desktop systems. Ncdu aims
to be fast, simple and easy to use, and should be able to run in any minimal
POSIX-like environment with ncurses installed.
=head2 Download
@ -14,13 +13,12 @@ to run in any minimal POSIX-like environment with ncurses installed.
=item Latest version
1.10 ([dllink ncdu-1.10.tar.gz download]
- L<changes|http://dev.yorhel.nl/ncdu/changes>
- L<mirror|https://sourceforge.net/projects/ncdu/files/ncdu/>)
1.11 ([dllink ncdu-1.11.tar.gz download]
- L<changes|https://dev.yorhel.nl/ncdu/changes>)
I also have convenient static binaries for Linux
L<i486|http://dev.yorhel.nl/download/ncdu-linux-i486-1.10.tar.gz> and
L<ARM|http://dev.yorhel.nl/download/ncdu-linux-arm-1.10.tar.gz>. Download,
L<i486|https://dev.yorhel.nl/download/ncdu-linux-i486-1.11.tar.gz> and
L<ARM|https://dev.yorhel.nl/download/ncdu-linux-arm-1.11.tar.gz>. Download,
extract and run; no compilation or installation necessary (uses
L<musl|http://www.musl-libc.org/>).
@ -34,9 +32,6 @@ L<online browsing|http://g.blicky.net/ncdu.git/>.
Ncdu is entirely written in C and available under a liberal MIT license.
Subscribe to L<freecode|http://freecode.com/projects/ncdu> to receive
notifications for new releases.
=head2 Packages and ports
@ -45,26 +40,25 @@ Ncdu has been packaged for quite a few systems, here's a list of the ones I am a
L<AgiliaLinux|http://packages.agilialinux.ru/search.php?tag=sys-fs> -
L<AIX|http://www.perzl.org/aix/index.php?n=Main.Ncdu> -
L<Alpine Linux|http://alpinelinux.org/packages?title_op=%3D&title=ncdu> -
L<ALT Linux|http://sisyphus.ru/en/srpm/Sisyphus/ncdu> -
L<Alpine Linux|http://pkgs.alpinelinux.org/package/main/x86/ncdu> -
L<ALT Linux|http://sisyphus.ru/en/srpm/ncdu> -
L<Arch Linux|http://www.archlinux.org/packages/?q=ncdu> -
L<CRUX|http://crux.nu/portdb/?q=ncdu&a=search> -
L<Cygwin|http://cygwin.com/cgi-bin2/package-grep.cgi?grep=ncdu> -
L<Debian|http://packages.debian.org/ncdu> -
L<Fedora|https://admin.fedoraproject.org/pkgdb/acls/name/ncdu> -
L<Fedora|https://admin.fedoraproject.org/pkgdb/package/ncdu/> -
L<FreeBSD|http://www.freshports.org/sysutils/ncdu/> -
L<Frugalware|http://frugalware.org/packages/?op=pkg&srch=ncdu&arch=all&ver=all> -
L<Gentoo|http://packages.gentoo.org/package/sys-fs/ncdu> -
L<GNU Guix|https://www.gnu.org/software/guix/package-list.html> -
L<IPCop|http://www.ipadd.de/binary-v2.html> -
L<OpenBSD|http://www.openbsd.org/cgi-bin/cvsweb/ports/sysutils/ncdu/> -
L<OpenBSD|http://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/sysutils/ncdu/> -
Mac OS X (L<Fink|http://pdb.finkproject.org/pdb/package.php/ncdu> - L<Homebrew|http://braumeister.org/formula/ncdu> - L<MacPorts|http://www.macports.org/ports.php?by=name&substr=ncdu>) -
L<Pardus|http://packages.pardus.org.tr/info/2011/testing/source/ncdu.html> -
L<Puppy Linux|http://www.murga-linux.com/puppy/viewtopic.php?t=35024> -
L<Solaris|http://www.opencsw.org/packages/ncdu> -
Slackware (L<Slackbuilds|http://slackbuilds.org/repository/14.1/system/ncdu/> - L<Slackers.it|http://www.slackers.it/repository/ncdu/>) -
L<Ubuntu|http://packages.ubuntu.com/search?searchon=sourcenames&keywords=ncdu> -
L<Zenwalk|http://zur.zenwalk.org/view/package/name/ncdu>
L<Slax Linux|http://www.slax.org/modules.php?detail=ncdu> -
L<Ubuntu|http://packages.ubuntu.com/search?searchon=sourcenames&keywords=ncdu>
Packages for CentOS, RHEL and (open)SUSE can be found on the
L<Open Build Service|https://build.opensuse.org/package/show?package=ncdu&project=utilities>.

6
dat/ncdu-changelog
View file

@ -1,3 +1,9 @@
1.11 - 2015-04-05
- Added 'b' key to spawn shell in the current directory
- Support scanning (and refreshing) of empty directories
- Added --si flag for base 10 prefixes
- Fix toggle dirs before files
1.10 - 2013-05-09
- Added 'c' key to display item counts
- Added 'C' key to order by item counts

14
dat/ncdu-jsonfmt
View file

@ -1,8 +1,8 @@
=pod
This document describes the file format that ncdu 1.9 and 1.10 uses for its
This document describes the file format that ncdu 1.9 and later use for the
export/import feature (the C<-o> and C<-f> options). Check the L<ncdu
manual|http://dev.yorhel.nl/ncdu/man> for a description on how to use that
manual|https://dev.yorhel.nl/ncdu/man> for a description on how to use that
feature.
=head2 Top-level object
@ -29,7 +29,7 @@ the existing format.
=head2 Metadata
The C<< <metadata> >> element is a JSON object holding whatever (short)
metadata you'd want. This block is currently (1.9-1.10) ignored by ncdu when
metadata you'd want. This block is currently (1.9-1.11) ignored by ncdu when
importing, but it writes out the following keys when exporting:
=over
@ -109,10 +109,10 @@ C<< 0 <= dev < 2^64 >>.
Number. Inode number as reported by C<lstat().st_ino>. Together with the Device
ID this uniquely identifies a file in this dump. In the case of hard links, two
objects may appear with the same (C<dev>,C<ino>) combination. A value of 0 is
assumed if this field is absent. This is currently (ncdu 1.9) not a problem as
long as the C<hlnkc> field is false, otherwise it will consider everything with
the same C<dev> and empty C<ino> values as a single hardlinked file. Accepted
values are in the range of C<< 0 <= ino < 2^64 >>.
assumed if this field is absent. This is currently (ncdu 1.9-1.11) not a
problem as long as the C<hlnkc> field is false, otherwise it will consider
everything with the same C<dev> and empty C<ino> values as a single hardlinked
file. Accepted values are in the range of C<< 0 <= ino < 2^64 >>.
=item hlnkc

16
dat/ncdu-man
View file

@ -97,6 +97,12 @@ option has no effect when C<-o> is used, because there will not be a browser
interface in that case. It has no effect when C<-f> is used, either, because
the deletion feature is disabled in that case anyway.
=item --si
List sizes using base 10 prefixes, that is, powers of 1000 (KB, MB, etc), as
defined in the International System of Units (SI), instead of the usual base 2
prefixes, that is, powers of 1024 (KiB, MiB, etc).
=back
=head2 Scan Options
@ -204,6 +210,14 @@ Show information about the current selected item.
Refresh/recalculate the current directory.
=item b
Spawn shell in current directory.
We first check the $SHELL environment variable of the user for the preferred
shell interpreter. If it's not set, we fall back to the compile time
configured default shell (usually /bin/bash).
=item q
Quit
@ -296,7 +310,7 @@ directory larger than 8 EiB minus one byte, ncdu will clip its size to 8 EiB
minus one byte.
Please report any other bugs you may find at the bug tracker, which can be
found on the web site at http://dev.yorhel.nl/ncdu
found on the web site at https://dev.yorhel.nl/ncdu
=head1 AUTHOR

2
dat/ncdu-scr
View file

@ -1,7 +1,7 @@
=pod
Note: While these screenshots are from version 1.7, the latest version has only
little visible changes. Let me also apologise for the crappy formatting, I
little visible changes. Let me also apologize for the crappy formatting, I
should take some smaller shots next time...
=head2 Scanning...

6
dat/sqlaccess
View file

@ -2,15 +2,15 @@ Multi-threaded Access to an SQLite3 Database
=pod
(Published on B<2011-11-26>. Also available in L<POD|http://dev.yorhel.nl/dat/sqlaccess>.)
(Published on B<2011-11-26>. Also available in L<POD|https://dev.yorhel.nl/dat/sqlaccess>.)
(Minor 2013-04-06 update: I abstracted my message passing solution from ncdc
and implemented it in a POSIX C library for general use. It's called
I<sqlasync> and is part of my L<Ylib library collection|http://dev.yorhel.nl/ylib>.)
I<sqlasync> and is part of my L<Ylib library collection|https://dev.yorhel.nl/ylib>.)
=head1 Introduction
As I was porting L<ncdc|http://dev.yorhel.nl/ncdc> over to use SQLite3 as
As I was porting L<ncdc|https://dev.yorhel.nl/ncdc> over to use SQLite3 as
storage backend, I stumbled on a problem: The program uses a few threads for
background jobs, and it would be nice to give these threads access to the
database.

6
dat/tuwf
View file

@ -39,7 +39,7 @@ information and details.
=head2 Download
B<Latest packaged version:> 0.2 ([dllink TUWF-0.2.tar.gz download]
B<Latest packaged version:> 1.0 ([dllink TUWF-1.0.tar.gz download]
- L<CPAN mirror|http://search.cpan.org/dist/TUWF/>)
TUWF is also available on a git repository at L<http://g.blicky.net/tuwf.git/>.
@ -55,11 +55,11 @@ TUWF is also available on a git repository at L<http://g.blicky.net/tuwf.git/>.
=item * L<Manned.org|http://manned.org/> (L<open source|http://g.blicky.net/manned.git/>)
=item * L<This website|http://dev.yorhel.nl/> (L<open source|http://g.blicky.net/yorhel-dev.git/tree/index.cgi>)
=item * L<This website|https://dev.yorhel.nl/> (L<open source|http://g.blicky.net/yorhel-dev.git/tree/index.cgi>)
=item * L<Blicky.net Pastebin|http://p.blicky.net/> (L<open source|http://g.blicky.net/bpaste.git/tree/index.cgi>)
=item * The website embedded in the L<D&R Axum|http://www.d-r.nl/AXUM/AXUM.htm> mixing console.
=item * The website embedded in the L<D&R Axum|http://www.d-r.nl/axum.html> mixing console.
=item * L<333networks|http://333networks.com/>

19
dat/tuwf-changelog
View file

@ -1,3 +1,22 @@
1.0 - 2015-09-17
- !! Some backwards-imcompatible changes, marked * !!
- kv_validate() improvements:
- Fix maxcount option
- Fix non-array argument to 'func'
- Added some default templates: num, int, uint, ascii, email, weburl
- * Removed 'min' and 'max' options, these now require the num template
- Add 'inherit' option for template definitions
- Allow templates to provide default values for 'required', 'default',
'rmwhitespace', 'multi', 'mincount' and 'maxcount'
- Add tests
- * reqPath() now includes the leading slash
- * reqGet(), reqPost(), reqParam(), reqUploadMIME() and reqUploadRaw()
now only work in scalar context.
- * Add plural versions of the above methods (reqGets() etc) that only
work in list context.
- Add reqQuery()
- Fix warning with Perl 5.22
0.2 - 2012-01-19
- Fixed bug with in-place utf8_decode() in recent Perls
- Lowered minimum Perl version to 5.8.0

2
dat/yxml
View file

@ -12,7 +12,7 @@ is available under a permissive MIT license. The only two files you need are
L<yxml.c|http://g.blicky.net/yxml.git/plain/yxml.c> and
L<yxml.h|http://g.blicky.net/yxml.git/plain/yxml.h>, which can easily be
included and compiled as part of your project. Complete API documentation is
available in L<the manual|http://dev.yorhel.nl/yxml/man>.
available in L<the manual|https://dev.yorhel.nl/yxml/man>.
The API follows a simple and mostly buffer-less design, and only consists of
three functions: