Lossy Logic Paradigm

Data compression algorithms fall into two categories: Lossless and Lossy. The former expects the decompressed data to be exactly the same as the original data while the latter can sacrifice a given amount of precision to achieve greater compression rates, so long as the decompressed data is Good Enough.

On the other hand, almost all other algorithms are lossless. For example, bigdatabasetable.getCount() to return exactly the correct, stable, no less of precision count at that time, no matter how many megabytes of information and seconds you need to run through to get that number. Which is annoying, when it returns 1e+6 when the context of the statement is:

if (bigdatabasetable.getCount() > 1024) {
    // Did we just scanned a terabyte of information? Oops.
    cout << "Database is not small!";
}

If you're using SQL, the above can slightly be "optimized" to

SELECT COUNT(*) FROM (SELECT 1 FROM bigdatabasetable LIMIT 1024) AS FOO

to avoid scanning too much information.

What'd be interesting is: bigdatabasetable.getEstimatedCount(100) where 100 would mean "spend up to about 100 msec on this, then give me your best result". In this case, we would be losing precision, but gaining control over the performance of the call.

getEstimatedCount is only a example of lossying a given algorithm or function call, but it can be equally applied to many other algorithms. bigarray.getBestMatch(".*?foobar", 100).getFirstTenItems().sort()

Applying Lossy Logic would mean we have to build applications to expect a little errors, but we do it all the time in time sensitive applications like video decoders and VOIP, network sensitive apps.

This coupled with Asynchronous calls via Message Bus might be an intriguing idea. Too sleepy to detail here now, but this'll allow us to easily make use of spare processors cores, either in the same computer, or nearby...

Asynchronous + Message Bus

Think Asynchronous, not Synchronous; Message Bus, not Point To Point.

It's a moment of epiphany when I realized the two related concepts are the key to many, many nagging software engineering problems of mine. Neither of them are alien to me, and I'm aware of them for quite a while. It's just that I've never realized how universal the patterns are, and how it applied to so many things. asdfe

Consider: how would you elegantly show the progress of a large file copy across several different user interfaces? (e.g. text, GUI, web?) It's tempting to get lost in low level details like Dependency Injection, when it's better to decouple the file copy process and the progress display into two asynchronous thread or process, and then send out periodic progress events to a Message Bus where interested user interfaces can display them. Caveat: this only applies to slow operations. This even allows for more than one user interface (listener) per operation. Traditionally, the file copy operation would be the same process that updates the User Interface's progress bar synchronously.

That was just the beginning as I found more and more problems can be elegantly solved by thinking asynchronously. AJAX web interfaces. dbus's interface between network events and the NetworkManager applet. Qt's Signals and Slots. Pipelining. Everyone of them are related.

Phrasing their definition to relate to each other:

Asynchronous: issue a request, but don't expect an immediate response. You'll get notified when the response is completed.

Message Bus: don't choose what component to talk to directly, instead send and forget to the bus (aka Message Queue, Channel, Slot), and other interested components themselves to choose to listen to the bus.

Vernor Vinge Motherlode

Finally found the rest of Vernor Vinge's works at the local book store today (Borders, Berjaya Times Square). Slurged on 4 of them: Tatja Grimm's World, Marooned in Realtime, Collected Stories of Vernor Vinge, The Peace War. Should be fun, Vinge's one of the few writers I dare to buy out right without research.

Adding to the pile, I got 3rd Ed. of Stevens' (and Fenner and Rudoff now) Unix Network Programming Volume 1. Had the hardcover 2nd ed, but don't know where that went. This new one is the more expensive (?!) softcover "International Edition".

'tis instant noodles for the forseeable future.