DataCleaner

A Web service was added to the monitoring application for getting a (list of) metric values. The 'Table lookup' component has been improved by adding join semantics as a configurable property. The EasyDQ components have been upgraded, adding further configuration options and a richer deduplication result interface. Performance improvements have been a specific focus of this release. Improvements have been made in the engine of DataCleaner to further utilize a streaming processing approach in certain corner cases which was not covered previously.

The date and time related analysis options have been expanded, adding distribution analyzers for week numbers, months, and years. An optional "descriptive statistics" option has been added to the Number analyzer and the Date/time analyzer The lines in the timeline charts of the monitoring Web application now have small dots in them. Two new transformers have been added for generating UUIDs and for generating timestamps. Now ad hoc queries can contain DISTINCT clauses, *-wildcards, and subqueries, and are fault-tolerant towards text-case issues.

Data Quality KPIs can now be defined as formulas (mathematical expressions), not just raw metrics.
It is now possible to fire ad-hoc SQL queries towards all datastores (DB, CSV, Excel, and more). A new analysis option, the Value matcher, was added. With this analysis, it's easy to identify unexpected values in a field. Management of jobs, including copying and deleting jobs, has been made a lot easier by exposing the functionality directly in the UI. It has been made possible to change historic data quality metrics in order to reposition results into the timeline.

This release adds minor bugfixes, performance improvements, and a few new features. Among the important ones are greatly-improved batch loading performance, a convenient "write data" menu in the main window, double-click renaming of job components, syntax coloring in the Javascript transformer and filter, and fixes for a potential deadlock when starting the application.

Support for MongoDB databases, both for read and write operations. Integration with EasyDQ.com, which provides Customer DQ functions in the cloud. Duplicate detection (aka. Deduplication / Fuzzy matching) analyzers. A "Table lookup" component for doing lookups of multiple values from a table. An "Insert into table" component for inserting records into any kind of table (e.g. database tables, CSV files, Excel sheets, or MongoDB collections). Job-level variables which allow for parameterizable jobs that can be instrumented from the command line.