vndb/yhdev
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Whole bunch of changes and releases

Browse source
This commit is contained in:
Yorhel 2018-01-29 14:32:46 +01:00
parent 07143cfb36
commit 54947f15c2
36 changed files with 3269 additions and 40 deletions
Download patch file Download diff file Expand all files Collapse all files

4
dat/doc
View file

@ -8,6 +8,10 @@ rare occasions are published on this page.
=over
=item C<2017-05-28 > - L<An Opinionated Survey of Functional Web Development|https://dev.yorhel.nl/doc/funcweb>
The title says it all.
=item C<2014-07-29 > - L<The Sorry State of Convenient IPC|https://dev.yorhel.nl/doc/easyipc>
A long rant about IPC systems.

571
dat/doc-funcweb Normal file
View file

@ -0,0 +1,571 @@
An Opinionated Survey of Functional Web Development
=pod
(Published on B<2017-05-28>.)
=head1 Intro
TL;DR: In this article I provide an overview of the frameworks and libraries
available for creating websites in statically-typed functional programming
languages.
I recommend you now skip directly to the next section, but if you're interested
in some context and don't mind a rant, feel free to read on. :-)
B<< <Rant mode> >>
When compared to native desktop application development, web development just
sucks. Native development is relatively simple with toolkits such as
L<Qt|https://www.qt.io/>, L<GTK+|https://www.gtk.org/> and others: You have
convenient widget libraries, and you can describe your entire application, from
interface design to all behavioural aspects, in a single programming language.
You're also largely free to structure code in whichever way makes most sense.
You can describe what a certain input field looks like, what happens when the
user interacts with it and what will happen with the input data, all succinctly
in a single file. There are even drag-and-drop UI builders to speed up
development.
Web development is the exact opposite of that. There are several different
technologies you're forced to work with even when creating the most mundane
website, and there's a necessary but annoying split between code that runs on
the server and code that runs in the browser. Creating a simple input field
requires you to consider and maintain several ends:
=over
=item
The back end (server-side code) that describes how the input field interacts
with the database.
=item
Some JavaScript code to describe how the user can interact with the input
field.
=item
Some CSS to describe what the input field looks like.
=item
And then there's HTML to act as a glue between the above.
=back
In many web development setups, all four of the above technologies are
maintained in different files. If you want to add, remove or modify an input
field, or just about anything else on a page, you'll be editing at least four
different files with different syntax and meaning. I don't know how other
developers deal with this, but the only way I've been able to keep these places
synchronized is to just edit one or two places, test if it works in a browser,
and then edit the other places accordingly to fix whatever issues I find. This
doesn't always work well: I don't get a warning if I remove an HTML element
somewhere and forget to also remove the associated CSS. Heck, in larger
projects I can't even tell whether it's safe to remove or edit a certain line
of CSS because I have no way to know for sure that it's not still being used
elsewhere. Perhaps this particular case can be solved with proper organization
and discipline, but similar problems exist with the other technologies.
Yet despite that, why do I still create websites in my free time? Because it is
the only environment with high portability and low friction - after all, pretty
much anyone can browse the web. I would not have been able to create a useful
"L<Visual Novel Database|https://vndb.org/>" any other way than through a
website. And the entire purpose of L<Manned.org|https://manned.org/> is to
provide quick access to man pages from anywhere, which is not easily possible
with native applications.
B<< </Rant mode> >>
Fortunately, I am not the only one who sees the problems with the "classic"
development strategy mentioned above. There are many existing attempts to
improve on that situation. A popular approach to simplify development is the
L<Single-page
application|https://en.wikipedia.org/wiki/Single-page_application> (SPA). The
idea is to move as much code as possible to the front end, and keep only a
minimal back end. Both the HTML and the entire behaviour of the page can be
defined in the same language and same file. With libraries such as
L<React|https://facebook.github.io/react/> and browser support for L<Web
components|https://developer.mozilla.org/en-US/docs/Web/Web_Components>, the
split between files described above can be largely eliminated. And if
JavaScript isn't your favorite language, there are many alternative languages
that compile to JavaScript. (See L<The JavaScript
Minefield|http://walkercoderanger.com/blog/2014/02/javascript-minefield/> for
an excellent series of articles on that topic).
While that approach certainly has the potential to make web development more
pleasant, it has a very significant drawback: Performance. For some
applications, such as web based email clients or CRM systems, it can be
perfectly acceptable to have a megabyte of JavaScript as part of the initial
page load. But for most other sites, such as this one, or the two sites I
mentioned earlier, or sites like Wikipedia, a slow initial page load is
something I consider to be absolutely unacceptable. The web can be really fast,
and developer laziness is not a valid excuse to ruin it. (If you haven't seen
or read L<The Website Obesity
Crisis|http://idlewords.com/talks/website_obesity.htm> yet, please do so now).
I'm much more interested in the opposite approach to SPA: Move as much code as
possible to the back end, and only send a minimal amount of JavaScript to the
browser. This is arguably how web development has always been done in the past,
and there's little reason to deviate from it. The difference, however, is that
people tend to expect much more "interactivity" from web sites nowadays, so the
amount of JavaScript is increasing. And that is alright, so long as the
JavaScript doesn't prevent the initial page from loading quickly. But this
increase in JavaScript does amplify the "multiple files" problem I ranted about
earlier.
So my ideal solution is a framework where I can describe all aspects of a site
in a single language, and organize the code among files in a way that makes
sense to me. That is, I want the same kind of freedom that I get with native
desktop software development. Such a framework should run on the back end, and
automatically generate efficient JavaScript and, optionally, CSS for the front
end. As an additional requirement (or rather, strong preference), all this
should be in a statically-typed language - because I am seemingly incapable of
writing large reliable applications with dynamic typing - and in a language
from functional heritage - because programming in functional languages has
spoiled me.
I'm confident that what I describe is possible, and it's evident that I'm not
the only person to want this, as several (potential) solutions like this do
indeed exist. I've been looking around for these solutions and have
experimented with a few that looked promising. This article provides an
overview of what I have found so far.
=head1 OCaml
My adventure began with L<OCaml|https://ocaml.org/>. It's been a few years
since I last used OCaml for anything, but development on the language and its
ecosystem have all but halted. L<Real World OCaml|https://realworldocaml.org/>
has been a great resource to get me up to speed again.
=head2 Ocsigen
For OCaml there is one project that has it all: L<Ocsigen|http://ocsigen.org/>.
It comes with an OCaml to JavaScript compiler, a web server, several handy
libraries, and a L<framework|http://ocsigen.org/eliom/> to put everything
together. Its L<syntax
extension|http://ocsigen.org/eliom/6.2/manual/ppx-syntax> allows you to mix
front and back end code, and you can easily share code between both ends. The
final result is a binary that runs the server and a JavaScript file that
handles everything on the client side.
The framework comes with an embedded DSL with which you can conveniently
generate HTML without actually typing HTML. And best of all, this DSL works on
both the client and the server: On the server side it generates an HTML string
that can be sent to the client, and running the same code on the client side
will result in a DOM element that is ready to be used.
Ocsigen makes heavy use of the OCaml type system to statically guarantee the
correctness of various aspects of the application. The HTML DSL ensures not
only that the generated HTML well-formed, but also prevents you from
incorrectly nesting certain elements and using the wrong attributes on the
wrong elements. Similarly, an HTML element generated on the server side can be
referenced from client side code without having to manually assign a unique ID
to the element. This prevents accidental typos in the ID naming and guarantees
that the element that the client side code refers to actually exists. URL
routing and links to internal pages are also checked at compile time.
Ocsigen almost exactly matches what I previously described as the perfect
development framework. Unfortunately, it has a few drawbacks:
=over
=item
The generated JavaScript is quite large, a bit over 400 KiB for an hello world.
In my brief experience with the framework, this also results in a noticeably
slower page load. I don't know if it was done for performance purposes, but
subsequent page views are per default performed via in-browser XHR requests,
which do not require that all the JavaScript is re-parsed and evaluated, and is
thus much faster. This, however, doesn't work well if the user opens pages in
multiple tabs or performs a page reload for whatever reason. And as I
mentioned, I care a lot about the initial page loading time.
=item
The framework has a steep learning curve, and the available documentation is by
far not complete enough to help you. I've found myself wondering many times how
I was supposed to use a certain API and have had to look for example code for
enlightenment. At some point I ended up just reading the source code instead of
going for the documentation. What doesn't help here is that, because of the
heavy use of the type system to ensure code correctness, most of the function
signatures are far from intuitive and are sometimes very hard to interpret.
This problem is made even worse with the generally unhelpful error messages
from the compiler. (A few months with L<Rust|https://www.rust-lang.org/> and
its excellent error messages has really spoiled me on this aspect, I suppose).
=item
I believe they went a bit too far with the compile-time verification of certain
correctness properties. Apart from making the framework harder to learn, it
also increases the verbosity of the code and removes a lot of flexibility. For
instance, in order for internal links to be checked, you have to declare your
URLs (or I<services>, as they call it) somewhere central such that the view
part of your application can access it. Then elsewhere you have to register a
handler to that service. This adds boilerplate and enforces a certain code
structure. And the gain of all this is, in my opinion, pretty small: In the 15
years that I have been building web sites, I don't remember a single occurrence
where I mistyped the URL in an internal link. I do suppose that this feature
makes it easy to change URLs without causing breakage, but there is a trivial
counter-argument to that: L<Cool URIs don't
change|https://www.w3.org/Provider/Style/URI.html>. (Also, somewhat ironically,
I have found more dead internal links on the Ocsigen website than on any other
site I have visited in the past year, so perhaps this was indeed a problem they
considered worth fixing. Too bad it didn't seem to work out so well for them).
=back
Despite these drawbacks, I am really impressed with what the Ocsigen project
has achieved, and it has set a high bar for the future frameworks that I will
be considering.
=head1 Haskell
I have always seen Haskell as that potentially awesome language that I just
can't seem to wrap my head around, despite several attempts in the past to
learn it. Apparently the only thing I was missing in those attempts was a
proper goal: When I finally started playing around with some web frameworks I
actually managed to get productive in Haskell with relative ease. What also
helped me this time was a practical introductory Haskell reference, L<What I
Wish I Knew When Learning Haskell|http://dev.stephendiehl.com/hask/>, in
addition to the more theoretical L<Learn You A Haskell for Great
Good|http://learnyouahaskell.com/>.
Haskell itself already has a few advantages when compared to OCaml: For one, it
has a larger ecosystem, so for any task you can think of there is probably
already at least one existing library. As an example, I was unable to find an
actively maintained SQL DSL for OCaml, while there are several available for
Haskell. Another advantage that I found where the much more friendly and
detailed error messages generated by the Haskell compiler, GHC. In terms of
build systems, Haskell has standardized on
L<Cabal|https://www.haskell.org/cabal/>, which works alright most of the time.
Packaging is still often complex and messy, but it's certainly improving as
L<Stack|http://haskellstack.org/> is gaining more widespread adoption. Finally,
I feel that the Haskell syntax is slightly less verbose, and more easily lends
itself to convenient DSLs.
Despite Haskell's larger web development community, I could not find a single
complete and integrated client/server development framework such as Ocsigen.
Instead, there are a whole bunch of different projects focussing on either the
back end or the front end. I'll explore some of them with the idea that,
perhaps, it's possible to mix and match different libraries and frameworks in
order to get the perfect development environment. And indeed, this seems to be
a common approach in many Haskell projects.
=head2 Server-side
Let's start with a few back end frameworks.
=over
=item Scotty
L<Scotty|https://github.com/scotty-web/scotty> is a web framework inspired by
L<Sinatra|http://www.sinatrarb.com/>. I have no experience with (web)
development in Ruby and have never used Sinatra, but it has some similarities
to what I have been using for a long time: L<TUWF|https://dev.yorhel.nl/tuwf>.
Scotty is a very minimalist framework; It does routing (that is, mapping URLs
to Haskell functions), it has some functions to access request data and some
functions to create and modify a response. That's it. No database handling,
session management, HTML generation, form handling or other niceties. But
that's alright, because there are many generic libraries to help you out there.
Thanks to its minimalism, I found Scotty to be very easy to learn and get used
to. Even as a Haskell newbie I had a simple website running within a day. The
documentation is appropriate, but the idiomatic way of combining Scotty with
other libraries is through the use of Monad Transformers, and a few more
examples in this area would certainly have helped.
=item Spock
Continuing with the Star Trek franchise, there's
L<Spock|https://www.spock.li/>. Spock is very similar to Scotty, but comes with
type-safe routing and various other goodies such as session and state
management, L<CSRF|https://en.wikipedia.org/wiki/Cross-site_request_forgery>
protection and database helpers.
As with everything that is (supposedly) more convenient, it also comes with a
slightly steeper learning curve. I haven't, for example, figured out yet how to
do regular expression based routing. I don't even know if that's still possible
in the latest version - the documentation isn't very clear. Likewise, it's
unclear to me what the session handling does exactly (Does it store something?
And where? Is there a timeout?) and how that interacts with CSRF protection.
Spock seems useful, but requires more than just a cursory glance.
=item Servant
L<Servant|http://haskell-servant.github.io/> is another minimalist web
framework, although it is primarily designed for creating RESTful APIs.
Servant distinguishes itself from Scotty and Spock by not only featuring
type-safe routing, it furthermore allows you to describe your complete public
API as a type, and get strongly typed responses for free. This also enables
support for automatically generated documentation and client-side API wrappers.
Servant would be an excellent back end for a SPA, but it does not seem like an
obvious approach to building regular websites.
=item Happstack / Snap / Yesod
L<Happstack|http://www.happstack.com/>, L<Yesod|http://www.yesodweb.com/> and
L<Snap|http://snapframework.com/> are three large frameworks with many
auxiliary libraries. They all come with a core web server, routing, state and
database management. Many of the libraries are not specific to the framework
and can be used together with other frameworks. I won't go into a detailed
comparison between the three projects because I have no personal experience
with any of them, and fortunately L<someone else already wrote a
comparison|http://softwaresimply.blogspot.nl/2012/04/hopefully-fair-and-useful-comparison-of.html>
in 2012 - though I don't know how accurate that still is today.
=back
So there are a fair amount of frameworks to choose from, and they can all work
together with other libraries to implement additional functions. Apart from the
framework, another important aspect of web development is how you generate the
HTML to send to the client. In true Haskell style, there are several answers.
For those who prefer embedded DSLs, there are
L<xhtml|http://hackage.haskell.org/package/xhtml>,
L<BlazeHTML|https://jaspervdj.be/blaze/> and
L<Lucid|https://github.com/chrisdone/lucid>. The xhtml package is not being
used much nowadays and has been superseded by BlazeHTML, which is both faster
and offers a more readable DSL using Haskell's do-notation. Lucid is heavily
inspired by Blaze, and attempts to L<fix several of its
shortcomings|http://chrisdone.com/posts/lucid>. Having used Lucid a bit myself,
I can attest that it is easy to get started with and pretty convenient in use.
I definitely prefer to generate HTML using DSLs as that keeps the entire
application in a single host language and with consistent syntax, but the
alternative approach, templating, is also fully supported in Haskell. The Snap
framework comes with L<Heist|https://github.com/snapframework/heist>, which are
run-time interpreted templates, like similar systems in most other languages.
Yesod comes with L<Shakespeare|http://hackage.haskell.org/package/shakespeare>,
which is a type-safe templating system with support for inlining the templates
in Haskell code. Interestingly, Shakespeare also has explicit support for
templating JavaScript code. Too bad that this doesn't take away the need to
write the JavaScript yourself, so I don't see how this is an improvement over
some other JavaScript solution that uses JSON for communication with the back
end.
=head2 Client-side
It is rather unusual to have multiple compiler implementations targeting
JavaScript for the same source language, but Haskell has three of them. All
three can be used to write front end code without touching a single line of
JavaScript, but there are large philosophical differences between the three
projects.
=over
=item Fay
L<Fay|https://github.com/faylang/fay/wiki> compiles Haskell code directly to
JavaScript. The main advantage of Fay is that it does not come with a large
runtime, resulting small and efficient JavaScript. The main downside is that it
only L<supports a subset of
Haskell|https://github.com/faylang/fay/wiki/What-is-not-supported?>. The
result is a development environment that is very browser-friendly, but where
you can't share much code between the front and back ends. You're basically
back to the separated front and back end situation in classic web development,
but at least you can use the same language for both - somewhat.
Fay itself doesn't come with many convenient UI libraries, but
L<Cinder|http://crooney.github.io/cinder/index.html> covers that with a
convenient HTML DSL and DOM manipulation library.
Fay is still seeing sporadic development activity, but there is not much of a
lively community around it. Most people have moved on to other solutions.
=item GHCJS
L<GHCJS|https://github.com/ghcjs/ghcjs> uses GHC itself to compile Haskell to a
low-level intermediate language, and then compiles that language to JavaScript.
This allows GHCJS to achieve excellent compatibility with native Haskell code,
but comes, quite predictably, at the high cost of duplicating a large part of
the Haskell runtime into the JavaScript output. The generated JavaScript code
is typically measured in megabytes rather than kilobytes, which is (in my
opinion) far too large for regular web sites. The upside of this high
compatibility, of course, is that you can re-use a lot of code between the
front and back ends, which will certainly make web development more tolerable.
The community around GHCJS seems to be more active than that of Fay. GHCJS
integrates properly with the Stack package manager, and there are a L<whole
bunch|http://hackage.haskell.org/packages/search?terms=ghcjs> of libraries
available.
=item Haste
L<Haste|https://github.com/valderman/haste-compiler> provides a middle ground
between Fay and GHCJS. Like GHCJS, Haste is based on GHC, but it instead of
using low-level compiler output, Haste uses a higher-level intermediate
language. This results in good compatibility with regular Haskell code while
keeping the output size in check. Haste has a JavaScript runtime of around 60
KiB and the compiled code is roughly as space-efficient as Fay.
While it should be possible to share a fair amount of code between the front
and back ends, not all libraries work well with Haste. I tried to use Lucid
within a Haste application, for example, but that did not work. Apparently one
of its dependencies (probably the UTF-8 codec, as far as I could debug the
problem) performs some low-level performance optimizations that are
incompatible with Haste.
Haste itself is still being sporadically developed, but not active enough to be
called alive. The compiler lags behind on the GHC version, and the upcoming 0.6
version has stayed unreleased and in limbo state for at least 4 months on the
git repository. The community around Haste is in a similar state. Various
libraries do exist, such as L<Shade|https://github.com/takeoutweight/shade>
(HTML DSL, Reactive UI), L<Perch|https://github.com/agocorona/haste-perch>
(another HTML DSL), L<haste-markup|https://github.com/ajnsit/haste-markup> (yet
another HTML DSL) and
L<haste-dome|https://github.com/wilfriedvanasten/haste-dome> (I<yet> another
HTML DSL), but they're all pretty much dead.
=back
Despite having three options available, only Haste provides enough benefit of
code reuse while remaining efficient enough for the kind of site that I
envision. Haste really deserves more love than it is currently getting.
=head2 More Haskell
In my quest for Haskell web development frameworks and tools, I came across a
few other interesting libraries. One of them is
L<Clay|http://fvisser.nl/clay/>, a CSS preprocessor as a DSL. This will by
itself not solve the CSS synchronisation problem that I mentioned at the start
of this article, but it could still be used to keep the CSS closer to code
implementing the rest of the site.
It also would not do to write an article on Haskell web development and not
mention a set of related projects: L<MFlow|https://github.com/agocorona/MFlow>,
L<HPlayground|https://github.com/agocorona/hplayground> and the more recent
L<Axiom|https://github.com/transient-haskell/axiom>. These are ambitious
efforts at building a very high-level and functional framework for both front
and back end web development. I haven't spend nearly enough time on these
projects to fully understand their scope, but I'm afraid of these being a bit
too high level. This invariably results in reduced flexibility (i.e. too many
opinions being hard-coded in the API) and less efficient JavaScript output.
Axiom being based on GHCJS reinforces the latter concern.
=head1 Other languages
I've covered OCaml and Haskell now, but there are relevant projects in other
languages, too:
=over
=item PureScript
L<PureScript|http://www.purescript.org/> is the spiritual successor of Fay -
except it does not try to be compatible with Haskell, and in fact
L<intentionally deviates from
Haskell|https://github.com/purescript/documentation/blob/master/language/Differences-from-Haskell.md>
at several points. Like Fay, and perhaps even more so, PureScript compiles down
to efficient and small JavaScript.
Being a not-quite-Haskell language, sharing code between a PureScript front end
and a Haskell back end is not possible, the differences are simply too large.
It is, however, possible to go into the other direction: PureScript could also
run on the back end in a NodeJS environment. I don't really know how well this
is supported by the language ecosystem, but I'm not sure I'm comfortable with
replacing the excellent quality of Haskell back end frameworks with a fragile
NodeJS back end (or such is my perception, I admittedly don't have too much
faith in most JavaScript-heavy projects).
The PureScript community is very active and many libraries are available in the
L<Persuit|https://pursuit.purescript.org/> package repository. Of note is
L<Halogen|https://pursuit.purescript.org/packages/purescript-halogen>, a
high-level reactive UI library. One thing to be aware of is that not all
libraries are written with space efficiency as their highest priority, the
simple L<Halogen
button|https://github.com/slamdata/purescript-halogen/tree/v2.0.1/examples/basic>
example already compiles down to a hefty 300 KB for me.
=item Elm
L<Elm|http://elm-lang.org/> is similar to PureScript, but rather than trying to
be a generic something-to-JavaScript compiler, Elm focuses exclusively on
providing a good environment to create web UIs. The reactive UI libraries are
well maintained and part of the core Elm project. Elm has a strong focus on
being easy to learn and comes with good documentation and many examples to get
started with.
=item Ur/Web
L<Ur/Web|http://www.impredicative.com/ur/> is an ML and Haskell inspired
programming language specifically designed for client/server programming. Based
on its description, Ur/Web is exactly the kind of thing I'm looking for: It
uses a single language for the front and back ends and provides convenient
methods for communication between the two.
This has been a low priority on my to-try list because it seems to be primarily
a one-man effort, and the ecosystem around it is pretty small. Using Ur/Web for
practical applications will likely involve writing your own libraries or
wrappers for many common tasks, such as for image manipulation or advanced text
processing. Nonetheless, I definitely should be giving this a try sometime.
(Besides, who still uses frames in this day and age? :-)
=item Opa
I'll be moving out of the functional programming world for a bit.
L<Opa|http://opalang.org/> is another language and environment designed for
client/server programming. Opa takes a similar approach to "everything in
PureScript": Just compile everything to JavaScript and run the server-side code
on NodeJS. The main difference with other to-JavaScript compilers is that Opa
supports mixing back end code with front end code, and it can automatically
figure out where the code should be run and how the back and front ends
communicate with each other.
Opa, as a language, is reminiscent of a statically-typed JavaScript with
various syntax extensions. While it does support SQL databases, its database
API seems to strongly favor object-oriented use rather than relational database
access.
=item GWT
Previously I compared web development to native GUI application development.
There is no reason why you can't directly apply native development structure
and strategies onto the web, and that's exactly what
L<GWT|http://www.gwtproject.org/> does. It provides a widget-based programming
environment that eventually runs on the server and compiles the client-side
part to JavaScript. I haven't really considered it further, as Java is not a
language I can be very productive in.
=item Webtoolkit
In the same vein, there's L<Wt|https://www.webtoolkit.eu/wt>. The name might
suggest that it is a web-based clone of Qt, and indeed that's what it looks
like. Wt is written in C++, but there are wrappers for L<other
languages|https://www.webtoolkit.eu/wt/other_language>. None of the languages
really interest me much, however.
That said, if I had to write a web UI for a resource-constrained device, this
seems like an excellent project to consider.
=back
=head1 To conclude
To be honest, I am a bit overwhelmed at the number of options. On the one hand,
it makes me very happy to see that a lot is happening in this world, and that
alternatives to boring web frameworks do exist. Yet after all this research I
still have no clue what I should use to develop my next website. I do like the
mix and match culture of Haskell, which has the potential to form a development
environment entirely to my own taste and with my own chosen trade-offs. On the
other hand, the client-side Haskell solutions are simply too immature and
integration with the back end frameworks is almost nonexistent.
Almost none of the frameworks I discussed attempt to tackle the CSS problem
that I mentioned in the introduction, so there is clearly room for more
research in this area.
There are a few technologies that I should spend more time on to familiarize
myself with. Ur/Web is an obvious candidate here, but perhaps it is possible to
create a Haskell interface to Wt. Or maybe some enhancements to the Haste
ecosystem could be enough to make that a workable solution instead.

2
dat/dump
View file

@ -27,7 +27,7 @@ Download: L<0.3|http://p.blicky.net/h25z8>
September 2011. L<ncdc|https://dev.yorhel.nl/ncdc> gained transfer logging
features, and I wrote a quick Perl script to fetch some simple statistics from
it. L<source|http://p.blicky.net/eu00a> (L<0.1|http://p.blicky.net/agolr>).
it. L<source|https://p.blicky.net/4V9Kg59kUJUN> (L<0.2|http://p.blicky.net/eu00a>, L<0.1|http://p.blicky.net/agolr>).
=head2 json.mll

9
dat/ncdc
View file

@ -49,10 +49,11 @@ L<Frugalware|http://frugalware.org/packages?srch=ncdc&op=pkg&arch=all&ver=all> -
L<Gentoo|http://packages.gentoo.org/package/net-p2p/ncdc> -
L<GNU Guix|https://www.gnu.org/software/guix/package-list.html> -
L<Homebrew|http://braumeister.org/formula/ncdc> -
L<OpenSUSE|http://packman.links2linux.org/package/ncdc>
L<OpenSUSE|http://packman.links2linux.org/package/ncdc> -
L<Source Mage|http://download.sourcemage.org/grimoire/codex/test/ftp/ncdc/>
I have a few old packages on the L<Open Build
Service|https://build.opensuse.org/package/show?package=ncdc&project=home%3Ayorhel>,
Service|https://build.opensuse.org/package/show/home:yorhel/ncdc>,
but these are unmaintained. The static binaries are preferred.
A convenient installer is available for
@ -60,10 +61,6 @@ L<Android|http://code.ivysaur.me/ncdcinstaller.html>.
=back
=cut
[html]<iframe src="http://software.opensuse.org/download/package.iframe?project=home:yorhel&package=ncdc" style="width: 560px; height: 200px; border: 1px dashed #aaa"></iframe>É
=head2 Features
Common features all modern DC clients (should) have:

10
dat/ncdu
View file

@ -13,12 +13,12 @@ POSIX-like environment with ncurses installed.
=item Latest version
1.12 ([dllink ncdu-1.12.tar.gz download]
1.13 ([dllink ncdu-1.13.tar.gz download]
- L<changes|https://dev.yorhel.nl/ncdu/changes>)
I also have convenient static binaries for Linux
L<i486|https://dev.yorhel.nl/download/ncdu-linux-i486-1.12.tar.gz> and
L<ARM|https://dev.yorhel.nl/download/ncdu-linux-arm-1.12.tar.gz>. Download,
L<i486|https://dev.yorhel.nl/download/ncdu-linux-i486-1.13.tar.gz> and
L<ARM|https://dev.yorhel.nl/download/ncdu-linux-arm-1.13.tar.gz>. Download,
extract and run; no compilation or installation necessary (uses
L<musl|http://www.musl-libc.org/>).
@ -65,12 +65,16 @@ L<Open Build Service|https://build.opensuse.org/package/show?package=ncdu&projec
Packages for NetBSD, DragonFlyBSD, MirBSD and others can be found on
L<pkgsrc|http://pkgsrc.se/sysutils/ncdu>.
A port to z/OS is available L<here|https://dovetail.com/community/ncdu.html>.
=head2 Similar projects
=over
=item L<Duc|http://duc.zevv.nl/> - Multiple user interfaces.
=item L<gt5|http://gt5.sourceforge.net/> - Quite similar to ncdu, but a different approach.
=item L<tdu|http://webonastick.com/tdu/> - Another small ncurses-based disk usage visualization utility.

9
dat/ncdu-changelog
View file

@ -1,3 +1,12 @@
1.13 - 2018-01-29
- Add "extended information" mode and -e flag
- Add file mode, modification time and uid/gid to info window with -e
- Add experimental color support and --color flag
- Add -rr option to disable shell spawning
- Remove directory nesting limit on file import
- Fix handling of interrupts during file import
- Fix undefined behavior that triggered crash on OS X
1.12 - 2016-08-24
- Add NCDU_SHELL environment variable
- Add --confirm-quit flag

48
dat/ncdu-jsonfmt
View file

@ -19,17 +19,18 @@ object is an array:
=head2 Versioning
The C<< <majorver> >> and C<< <minorver> >> elements indicate the version of the file
format. These are numbers with accepted values in the range of
C<< 0 <= version <= 10000 >>. Major version must be C<1>, minor version is currently C<0>. The
major version should increase if backwards-incompatible changes are made
(preferably never), the minor version can be increased to indicate additions to
the existing format.
The C<< <majorver> >> and C<< <minorver> >> elements indicate the version of
the file format. These are numbers with accepted values in the range of C<< 0
<= version <= 10000 >>. Major version must be C<1>. Minor version is C<0> for
ncdu 1.9 till 1.12, and C<1> since ncdu 1.13 for the addition of the extended
mode. The major version should increase if backwards-incompatible changes are
made (preferably never), the minor version can be increased to indicate
additions to the existing format.
=head2 Metadata
The C<< <metadata> >> element is a JSON object holding whatever (short)
metadata you'd want. This block is currently (1.9-1.12) ignored by ncdu when
metadata you'd want. This block is currently (1.9-1.13) ignored by ncdu when
importing, but it writes out the following keys when exporting:
=over
@ -109,7 +110,7 @@ C<< 0 <= dev < 2^64 >>.
Number. Inode number as reported by C<lstat().st_ino>. Together with the Device
ID this uniquely identifies a file in this dump. In the case of hard links, two
objects may appear with the same (C<dev>,C<ino>) combination. A value of 0 is
assumed if this field is absent. This is currently (ncdu 1.9-1.12) not a
assumed if this field is absent. This is currently (ncdu 1.9-1.13) not a
problem as long as the C<hlnkc> field is false, otherwise it will consider
everything with the same C<dev> and empty C<ino> values as a single hardlinked
file. Accepted values are in the range of C<< 0 <= ino < 2^64 >>.
@ -163,6 +164,37 @@ whatever else your system may support. If absent, C<false> is assumed.
=back
=head3 Extended information
In addition, the following fields are exported when I<extended information>
mode is enabled (available since ncdu 1.13). See the C<-e> flag in L<ncdu(1)>
for details.
=over
=item uid
Number, user ID who owns the file. Accepted values are in the range
C<< 0 <= uid < 2^31 >>.
=item gid
Number, group ID who owns the file. Accepted values are in the range
C<< 0 <= uid < 2^31 >>.
=item mode
Number, the raw file mode as returned by L<lstat(3)>. For Linux systems, see
L<inode(7)> for the interpretation of this field. Accepted range:
C<< 0 <= mode < 2^16 >>.
=item mtime
Number, last modification time as a UNIX timestamp. Accepted range:
C<< 0 <= mtime < 2^64 >>.
=back
=head2 Miscellaneous notes
As mentioned above, file/directory names are B<not> converted to any specific

33
dat/ncdu-man
View file

@ -35,8 +35,8 @@ I<FILE> is equivalent to C<->, the file is read from standard input.
For the sake of preventing a screw-up, the current version of ncdu will assume
that the directory information in the imported file does not represent the
filesystem on which the file is being imported. That is, the refresh and file
deletion options in the browser will be disabled.
filesystem on which the file is being imported. That is, the refresh, file
deletion and shell spawning options in the browser will be disabled.
=item I<dir>
@ -54,6 +54,19 @@ directory with many files. 10.000 files will get you an export in the order of
gzip. This scales linearly, so be prepared to handle a few tens of megabytes
when dealing with millions of files.
=item -e
Enable extended information mode. This will, in addition to the usual file
information, also read the ownership, permissions and last modification time
for each file. This will result in higher memory usage (by roughly ~30%) and in
a larger output file when exporting.
When using the file export/import function, this flag will need to be added
both when exporting (to make sure the information is added to the export), and
when importing (to read this extra information in memory). This flag has no
effect when importing a file that has been exported without the extended
information.
=back
=head2 Interface options
@ -97,6 +110,16 @@ option has no effect when C<-o> is used, because there will not be a browser
interface in that case. It has no effect when C<-f> is used, either, because
the deletion feature is disabled in that case anyway.
WARNING: This option will only prevent deletion through the file browser. It is
still possible to spawn a shell from ncdu and delete or modify files from
there. To disable that feature as well, pass the C<-r> option twice (see
C<-rr>).
=item -rr
In addition to C<-r>, this will also disable the shell spawning feature of the
file browser.
=item --si
List sizes using base 10 prefixes, that is, powers of 1000 (KB, MB, etc), as
@ -108,6 +131,12 @@ prefixes, that is, powers of 1024 (KiB, MiB, etc).
Requires a confirmation before quitting ncdu. Very helpful when you
accidentally press 'q' during or after a very long scan.
=item --color I<SCHEME>
Select a color scheme. Currently only two schemes are recognized: I<off> to
disable colors (the default) and I<dark> for a color scheme intended for dark
backgrounds.
=back
=head2 Scan Options

18
dat/ncdu-scr
View file

@ -1,29 +1,29 @@
=pod
Note: While these screenshots are from version 1.7, the latest version has only
little visible changes. Let me also apologize for the crappy formatting, I
should take some smaller shots next time...
These screenshots were made with ncdu 1.13 with the C<--color=dark> option.
Colors are not available in older versions and (in 1.13) still disabled by
default.
=head2 Scanning...
[img scr ncduscan.png Ncdu scanning a large directory.]
[img scr ncduscan-2.png Ncdu scanning a large directory.]
=head2 Done scanning
[img scr ncdudone.png Ncdu done scanning a large directory.]
[img scr ncdudone-2.png Ncdu done scanning a large directory.]
=head2 Directory information
[img scr ncduinfo.png Ncdu displaying directory information.]
[img scr ncduinfo-2.png Ncdu displaying directory information.]
=head2 Delete confirmation
[img scr ncduconfirm.png Ncdu asking for confirmation to delete a file.]
[img scr ncduconfirm-2.png Ncdu asking for confirmation to delete a file.]
=head2 Help screen
[img scr ncduhelp1.png Ncdu help screen.]
[img scr ncduhelp1-2.png Ncdu help screen.]
=head2 About screen
[img scr ncduhelp2.png Ncdu about screen.]
[img scr ncduhelp2-2.png Ncdu about screen.]

54
dat/nginx-confgen Normal file
View file

@ -0,0 +1,54 @@
=pod
nginx-confgen is a simple preprocessor and macro system for
L<nginx|http://nginx.org/> and nginx-like configuration files. It support
variable substitution, macro expansion and using the output of arbitrary
commands to generate config files.
=head2 Example
pre_set $certdir /etc/nginx-certificates/;
# Fetch the 'resolver' from /etc/resolv.conf
pre_exec $nameserver "grep nameserver /etc/resolv.conf \\
| head -n 1 | sed 's/^nameserver //'";
resolver $nameserver;
# Convenient macro to create a HTTPS virtual host
macro vhost $domain @aliases &block {
server {
listen [::]:443 ssl;
server_name $domain @aliases;
ssl_certificate $certdir/$domain/fullchain.pem;
ssl_certificate_key $certdir/$domain/privkey.pem;
pre_if -f $certdir/$domain/ocsp.der {
ssl_stapling_file $certdir/$domain/ocsp.der;
}
&block;
}
}
vhost example.com www.example.com {
root /var/www/example.com;
}
See the L<manual|https://dev.yorhel.nl/nginx-confgen/man> for more features.
=head2 Download
If you're on a x86_64 Linux system, you can simply use the binary:
curl -s https://dev.yorhel.nl/download/nginx-confgen-linux-amd64-1.1.tar.gz | tar -xzf-
./nginx-confgen <input.conf >output.conf
To compile from source, install L<Haskell Stack|https://haskellstack.org/> and run:
git clone git://g.blicky.net/nginx-confgen.git
cd nginx-confgen
stack install
The git repository is also available for L<online
browsing|https://g.blicky.net/nginx-confgen.git/>.

8
dat/nginx-confgen-changelog Normal file
View file

@ -0,0 +1,8 @@
1.1 - 2018-01-24
- Add pre_warn directive
- Add -i/-o/-v/-h command line arguments
- Add support for custom pre_include search paths (-I flag)
- Fix handling of some common custom block directives (e.g. 'types')
1.0 - 2018-01-19
- Initial version

285
dat/nginx-confgen-man Normal file
View file

@ -0,0 +1,285 @@
=pod
=head1 NAME
nginx-confgen - A preprocessor and macro system for nginx(-like) configuration
files.
=head1 SYNOPSIS
nginx-confgen -i input.conf -o output.conf
=head1 DESCRIPTION
nginx-confgen can be used to do pre-processing for nginx configuration files
(and other configuration files with a similar syntax). It has support for
"compile-time" macro expansion and variable interpolation, which should make it
less tedious to maintain large and complex configurations.
nginx-confgen does not currently support any command-line arguments. It simply
reads the configuration from standard input, and writes the processed
configuration to standard output.
nginx-confgen works by parsing the input into a syntax tree, modifying this
tree, and then formatting the tree to generate the output. It is completely
oblivious to nginx contexts and directives, so it is possible to do nonsensical
transformations and generate incorrect configuration files. Comments in the
input file will not be present in the output. See also the L</BUGS & WARTS>
below.
B<WARNING:> Do NOT use nginx-confgen with untrusted input, the C<pre_exec>
directive allows, by design, arbitrary code execution.
=head1 OPTIONS
The following command-line options are supported:
=over
=item -h
Show help text.
=item -V, --version
Show program version.
=item -i I<FILE>
Use the given file name as input file. If this option is not given or set to
C<->, then the file will be read from standard input.
=item -o I<FILE>
Write the output to the given file. If this option is not given or set to C<->,
then the file will be written to standard output.
=item -I I<DIR>
Set the search path for I<pre_include> directives. This option can be given
multiple times to search several directories in order. If this option is not
given, then include files are resolved relative to the directory that
nginx-confgen is run from (i.e. C<-I .>).
=back
=head1 DIRECTIVES
nginx-confgen recognizes and interprets the following directives:
=head2 pre_include
Similar to the C<include> directive in nginx, except that the file is included
during preprocessing. The included file may contain any preprocessing
directives supported by nginx-confgen. Variables and macros defined in the
included file will be available in the parent file.
Relative paths are searched for in the directories given with the C<-I> flag.
=head2 pre_set
Similar to the C<set> directive in nginx, except that variables defined with
C<pre_set> are resolved during preprocessing. Note that variables defined with
C<pre_set> are only available in the same scope as they are defined, for
example:
pre_set $var outer;
location / {
pre_set $var inner;
# $var is now "inner" within this location block.
}
# $var is "outer" again after the location block.
(This may change in the future)
=head2 pre_exec
Run a shell command, and store the output in a variable. For example, nginx
will not use your system's DNS resolution methods to resolve domain names.
Instead you need to manually set a C<resolver> address. With the following hack
you can fetch the nameserver from C</etc/resolv.conf> and use that as the
C<resolver>:
pre_exec $nameserver "grep nameserver /etc/resolv.conf \\
| head -n 1 | sed 's/^nameserver //'";
resolver $nameserver;
(The C<\\> is necessary, otherwise your shell will consider the newline as a
new command).
=head2 pre_if
Similar to the C<if> directive in nginx, except that this is evaluated during
preprocessing. nginx-confgen has a few warts with regards to parenthesis,
things usually work better without:
pre_if -f $certdir/ocsp.der {
ssl_stapling on;
ssl_stapling_file $certdir/ocsp.der;
}
pre_if !-f $certdir/ocsp.der {
ssl_stapling off;
}
# You can have different configuration depending on the name of
# the system on which nginx-confgen runs. Like... yeah.
pre_exec $hostname 'hostname';
pre_if $hostname ~* ^proxy_for_(.+) {
proxy_pass http://$1/;
}
=head2 pre_warn
This directive, when interpreted, will generate a warning to the standard error
of nginx-confgen. Can be used to signal that a special configuration is being
used:
pre_if -e /etc/offline-mode {
pre_warn "Putting website in offline mode!";
}
Or to warn about certain directives:
pre_macro proxy_cache $var {
pre_warn "Using proxy_cache with $var violates company policy!";
# But we can output it anyway.
proxy_cache $var;
}
=head2 macro
Define a I<macro>, which is a configuration block that you can later refer to.
The general syntax is as follows:
macro macro_name $var1 $var2 @remaining_vars &block_var {
# contents
}
The optional C<@remaining_vars> argument will capture any number of variables,
and can be passed to another directive inside the macro contents. The optional
C<&block_var> allows the macro to be invoked with a block argument, which will
expand to any number of directives. Some examples:
macro le {
location /.well-known/acme-challenge {
alias /etc/letsencrypt/challenge;
}
}
# Usage:
le;
macro redir $path $to {
location $path {
return 301 $to;
}
}
# Usage:
redir / http://blicky.net/;
macro vhost $primary_name @aliases &block {
server {
listen [::]:443 ssl;
server_name $primary_name @aliases;
ssl_certificate $crtdir/$primary_name/fullchain.pem;
ssl_certificate_key $crtdir/$primary_name/privkey.pem;
&block;
}
}
# Usage:
vhost example.com {
root /var/www/example.com;
}
vhost example.org alias.example.org {
root /var/www/example.org;
}
Note that these are I<hygienic> macros, so variable capture is predictable (but
not necessarily the most useful):
pre_var $dest /a;
macro redir {
# This will be /a, regardless of the context in which this macro is called.
return 301 $dest;
}
# $dest is still '/a' inside the macro after this new variable definition.
pre_var $dest /b;
redir;
Similarly, macro arguments will not be available inside C<&block> expansion or
nested macro expansion.
=head1 BUGS & WARTS
nginx-confgen is a quickly written hack to solve a particular use case, it is
quite likely to have some weird behavior and bugs. Here's a few I am aware of:
=over
=item *
The nginx configuration syntax is not as regular as I had hoped. It's possible
for nginx modules to extend the syntax somewhat. A good example is the I<types>
directive in I<ngx_http_core_module>. While nginx-confgen should be able to
handle the I<types> directive just fine, other extensions may cause syntax
errors or will not survive a round-trip through nginx-confgen.
This applies to all I<*_by_lua_block> directives in the I<ngx_http_lua_module>.
The I<_by_lua> directives that accept a string should work just fine.
=item *
Be careful with parenthesis around if statements, e.g.:
if ($a == $b) { }
Will get converted into:
if "(${a}" == "${b})" { }
Which is unlikely what you want. As a workaround, add some spaces:
if ( $a == $b ) { }
=item *
Arguments to directives may get reformatted, especially if they contain a
variable. This I<should> not matter in most cases, but in some particular
scenarios it does. Here's a few examples of reformatting:
return 301 http://blicky.net$request_uri;
# becomes:
return 301 "http://blicky.net${request_uri}";
add_header Something "${header}";
# becomes:
add_header Something $header;
This reformatting may cause different behavior for nginx directives that do not
support variable interpolation, such as C<error_log>.
=item *
C<pre_if> does not like empty strings, e.g.
pre_if $x == "" { }
Will throw an error, use the following instead:
pre_if $x { }
=item *
The error messages aren't always helpful.
=back
=head1 AUTHOR
nginx-confgen is written by Yoran Heling <projects@yorhel.nl>
Web: L<https://dev.yorhel.nl/nginx-confgen>

12
dat/tuwf
View file

@ -39,10 +39,10 @@ information and details.
=head2 Download
B<Latest packaged version:> 1.0 ([dllink TUWF-1.0.tar.gz download]
B<Latest packaged version:> 1.1 ([dllink TUWF-1.1.tar.gz download]
- L<CPAN mirror|http://search.cpan.org/dist/TUWF/>)
TUWF is also available on a git repository at L<http://g.blicky.net/tuwf.git/>.
TUWF is also available on a git repository at L<https://g.blicky.net/tuwf.git/>.
=head2 Websites using TUWF
@ -51,13 +51,13 @@ TUWF is also available on a git repository at L<http://g.blicky.net/tuwf.git/>.
=over
=item * L<VNDB.org|http://vndb.org/> (the site that spawned TUWF - L<open source|http://g.blicky.net/vndb.git/>)
=item * L<VNDB.org|https://vndb.org/> (the site that spawned TUWF - L<open source|https://g.blicky.net/vndb.git/>)
=item * L<Manned.org|http://manned.org/> (L<open source|http://g.blicky.net/manned.git/>)
=item * L<Manned.org|https://manned.org/> (L<open source|https://g.blicky.net/manned.git/>)
=item * L<This website|https://dev.yorhel.nl/> (L<open source|http://g.blicky.net/yorhel-dev.git/tree/index.cgi>)
=item * L<This website|https://dev.yorhel.nl/> (L<open source|https://g.blicky.net/yorhel-dev.git/tree/index.cgi>)
=item * L<Blicky.net Pastebin|http://p.blicky.net/> (L<open source|http://g.blicky.net/bpaste.git/tree/index.cgi>)
=item * L<Blicky.net Pastebin|https://p.blicky.net/> (L<open source|https://g.blicky.net/bpaste.git/tree/index.cgi>)
=item * The website embedded in the L<D&R Axum|http://www.d-r.nl/axum.html> mixing console.

7
dat/tuwf-changelog
View file

@ -1,3 +1,10 @@
1.1 - 2017-11-26
- Disallow exclamation mark in email address validation
- Add reqProtocol() method
- Add reqFCGI() method
- Remove 'X-Powered-By' header
- Fix handling of space character in load_recursive()
1.0 - 2015-09-17
- !! Some backwards-imcompatible changes, marked * !!
- kv_validate() improvements: