Integrating Teradata ADS Generator with Fuzzy Logix DB Lytix

Short teaser:

ADS Generator and DBLytix offer an extremely powerful in-database predictive analytics platform

Attachment	Size
Teradata Warehouse Miner Project File for Fuzzy Logix Integration	158.83 KB

Teradata ADS Generator, part of the Teradata Warehouse Miner family of products, was built to support both comprehensive data profiling as well as analytic data generation for Teradata customers. Neither the data profiling nor the analytic data set generation capabilities of the product require any movement of data outside of the warehouse and utilize as much of the data as the analyst or data scientist requires. Results and metadata are stored directly in the database while utilizing the parallel, scalable processing power of the Teradata platforms to perform data intensive operations.

DB Lytix^™, a product developed by Fuzzy Logix, offers scalable and robust high performance analytical methods that are embedded seamlessly into database systems. The DB Lytix^™ library of statistical, machine learning, and quantitative methods provide Teradata customers a rich set of in-database components which fall into one of the following categories:

Cumulative Distribution Functions (40)	Data Mining Functions (66)
Date Functions (26)	Fit Distributions (78)
Hypothesis Testing Functions (33)	Inverse Cumulative Distribution Functions (40)
Mathematical Functions (23)	Probability Density Functions (40)
Sampling Techniques Functions (5)	Simulate Copula Functions (7)
Simulate Univariate Functions (40)	Sparse Statistics Functions (11)
Statistical Functions (50)	String Functions (14)
Time Series Functions (6)

The Fuzzy Logix DB Lytix^™ library of advanced analytic functions can be deployed on Teradata 13.10, Teradata 14.00, and Teradata 14.10 as embedded Teradata database objects in the form of User Defined Functions (UDF’s) and External Stored Procedures (XSP’s) as follows:

Scalar UDF’s - for those analytics that require a single result / single row as an answer set.
Aggregate UDF’s - for those analytics that require single result of multiple aggregated rows as an answer set.
Table UDF’s - for those analytics that require a table of results as an answer set.
XSP’s - for those analytics that create multiple results and/or iterate over the input data multiple times. The XSP controls the iteration of the SQL executed, often with DB Lytix^™ embedded UDF’s in it.

Recently, Teradata’s Partner Integration Lab (PIL) worked closely with Fuzzy Logix to port their DB Lytix^™ advanced analytics library to the Teradata platform. At the same time PIL collaborated with the Teradata Applications Engineering (TAE) team to integrate the Teradata ADS Generator product with DB Lytix^™. The integration of these two products provide an extremely powerful in-database predictive analytics platform providing data profiling, data preparation, machine learning, multivariate statistics and predictive model deployment capabilities for Teradata customers.

As a result of the collaboration, recent changes were made by TAE to the Teradata Warehouse Miner family of products, allowing the Teradata ADS Generator front-end to call the DB Lytix^™ database objects, and manage them for production environments. Teradata Warehouse Miner or ADS Generator Version 05.03.05 is required for this integration. The combined ADS Generator/DB Lytix^™ integrated architecture is depicted below:

In this depiction, XSP’s run at the Parsing Engine level, while UDF’s run at the AMP level. Since there are over 500 (and counting!) DB Lytix^™ functions within their library, it is not possible to illustrate how each would be called and managed from Teradata ADS Generator. What we have done though, is to put together a tutorial on how to call each type of DB Lytix^™ function, and how to use the Teradata ADS Generator’s Variable Creation Analysis to manage their execution and interrogate the results of their in-database analysis. This tutorial is available for download here:

Teradata ADS Generator & Fuzzy Logix DB Lytix^™ Integration Guide

This guide describes the integration points between the two products and does not attempt to replicate the information in either the Teradata ADS Generator or Fuzzy Logix DB Lytix^™ user documentation. For more information on the capabilities of Teradata ADS Generator, please refer to the following:

For a thorough description of all the functions available in the Fuzzy Logix DB Lytix^™ library, please refer to the following:

User Manual for DB Lytix^™on Teradata Advanced Package v1.0.1

The Teradata ADS Generator & Fuzzy Logix DB Lytix^™ Integration Guide is meant to provide a functional description on how each type of database object available within the Fuzzy Logix DB Lytix^™library is called from the Teradata ADS Generator User Interface. It also includes an example of one of the Fuzzy Logix Excel Templates that is available for visualizing the results generated by their database objects. A thorough description of all the Excel Templates in a use-case oriented format is available in the following companion document:

Teradata ADS Generator - User Guide for Integrating DB Lytix^™and Excel (Aug 2014)

For simplicity, the Teradata ADS Generator & Fuzzy Logix DB Lytix^™ Integration Guide utilizes the well-known TWM Demonstration Data that is released with each version of Teradata Warehouse Miner or Teradata ADS Generator. Refer to Volume 1 of the TWM User Guide listed above for instructions on how to install this data. Attached to this reference is a Zip file containing the supporting ADS Project “.bin” file that can be imported into your Teradata ADS Generator environment. Explicit instructions for importing the “Teradata ADS Demo Powered by Fuzzy Logix.bin” file are included in the Teradata ADS Generator & Fuzzy Logix DB Lytix^™ Integration Guide.

The Teradata ADS Generator & Fuzzy Logix DB Lytix^™ Integration Guide describes in step-by-step fashion, how to build out the analyses within the “Teradata ADS Demo Powered by Fuzzy Logix” ADS Project. There are analyses which support the detailed descriptions in the Integration Guide for each of the following:

Creating Variables by Executing DB Lytix^™ Scalar and Aggregate UDF’s
Creating Variables From DB Lytix^™ Table UDF’s
Calling Fuzzy Logix XSP’s from Variable Creation

The Integration Guide also introduces the enabling technology within Teradata ADS Generator required for the DB Lytix^™ integration, namely the Variable Creation analysis. In specific, the unique features of Variable Creation that enable the integration are Run Units, Literals and Literal Parameters. Also included is an example of using Run Units to launch Microsoft Excel to visualize Fuzzy Logix results. A sample screen capture illustrating the integration of Fuzzy Logix, Teradata ADS Generator as well as the Fuzzy Logix Excel templates is shown below:

As of this publication, additional integration features have been added to Teradata Warehouse Miner and ADS Generator version 5.3.5.1. These features are documented within the Help system available with the product, but not within the documents listed above. Most of the examples within the Integration Guide can be performed without the 5.3.5.1 patch. Discrepancies that you may notice include the following:

Issue Type	Summary
Improvement	Allow dropping column names or values on Text or String Literal or Parameter properties text box.
Improvement	Allow the same Text or String Parameter to be enclosed in single quotes or not so for different occurrences.
New Feature	Allow placeholders for Project Attachment and Data Source Name in Program Execution fields
Improvement	Add option to Program Execution run unit to run in a detached process (don't wait)
New Feature	Add "template" selector to Add New Analysis dialog so that the "shell" of a complex analysis may be created
Improvement	Allow dropping values or columns on String literals or parameters as done with Text literals or parameters
Improvement	Allow setting or adding environment variables when executing a script or program
Improvement	Allow setting of the Working Directory when executing a script or program.

In order to obtain TWM / ADS or Fuzzy Logix Software, along with the 5.3.5.1 patch, the following links are provided:

Teradata Associates

TWM or ADS software for version 5.3.5, as well as the 5.3.5.1 patch, can be downloaded from the Teradata Warehouse Miner Sharepoint site located at:

http://sharepoint.teradata.com/twm

An evaluation version of Fuzzy Logix software and the documentation listed aboce can be obtained sending an E-Mail to downloads@fuzzyl.com with a with a cc: Klepacz, Christopher with the following information: