system development

KaleidOk started out as a collection of Processing sketches and later plain Java applications based on the Processing core library. They served as design concepts and technical proofs-of-concept for the various data sources, of which kaleidOk makes use; audio analysis, speech recognition, emotional text analysis and colour-based image search, and to familiarise ourselves with the involved libraries and APIs. We used the source code revision control system Git hosted by GitHub to track changes from very early on in the development phase. Shortly thereafter, we began to use the issue tracking system provided by GitHub and the integrated development environment IntelliJ IDEA.

In May 2015, we integrated the code and functionality of the individual proofs-of-concept into modules of a single application dubbed “Kaleidoscope”. The incompatibilities between some of the employed software libraries uncovered during integration process led to their substitution by or supplementation with a similar library Minim vs. TarsosDSP or a nearly complete rewrite of Speech-to-Text.

It then became apparent to us that a single-threaded, serial execution of the graphics renderer and requests to network services Google Speech API, Chromatik and Flickr are not feasible for usability in the long run, this led to the conversion of network requests to task objects, that would be handled by concurrently running worker threads. This execution is incompatible with the service connector libraries and required their partial re-implementation.

Since an increasing amount of sketch parameters are subject to frequent changes, it became necessary to extend Processing’s sketch framework, which is based on Java Applets, to support run time configuration. Other necessary extensions included setting up the applet container to fit presentation needs better, most importantly a flexible full screen display of the sketch as well as graphical user-interface elements like text fields and buttons.

kaleidOk is deployed as a Java application with all the necessary library dependencies most importantly Synesketch and also Apache Commons, Gson, javaFlacEncoder, DiskLRUCache, and Wiremock. The computer connects to an external display (a monitor, large screen or a projector) and a microphone. A network connection is required to allow access to media indices and archives and optionally other KaleidOk deployments of collaborating participants. To reduce the amount of network traffic and the waiting time for the display of images, we integrated a cache for HTTP requests.

One important requirement of further research is the documentation of test results, which led to us integrating a tuneable, structured logging facility into kaleidOk. On top of textual logs, kaleidOk now stores local copies of the audio records used for speech recognition and screen shots of the sketch for every new image appearing in it.