Accessing Historical and Current Data with Unity Director

Unity Director is an extremely capable product that offers a wide variety of benefits. One of it's unique benefits is the ability to easily route users and requests to specific Teradata systems. There are many potential uses of this ability; but one use, in particular, is to selectively direct users that want access to historical data to specific systems where the data resides. 

However, you don't need to wait until Unity Director 14.10 to start using this functionality. There is a way to leverage different table-depths in the current release of the product.  Here’s an example of how you can start creating system specific views today with Unity Director 14.00.

Creating the history and current data views

The follow example illustrates how to create a set of views, a currentYearView to display data from the last year only, and a historyView to display data from all the previous years. While the data supporting the currentYearView is kept on both systems (system 1 & system 2), the data for the historyView is retained only on system 2.

1.       The base data table, which will contain the current and historical records, is created on both systems:

create table basedata (id integer, ts timestamp);

2.       On both systems, create a current view (currentYearView) that will return only current data (in this case the rows from the last year).

create view currentYearView as
 locking basedata for access
 select * from basedata where ts between '2013-01-01 00:00:00' and '2013-12-31 23:59:59

3.       An additional table, called system2Only is also created only on system 2.This will help direct queries to the historical data on system 2:

create table system2only (id integer); -- And Empty table, with arbitrary columns

4.       Also on system 2, a history view (historyView) is created that joins to the system2Only table:

create view historyView as
 locking basedata for access
 select * from basedata
 full outer join system2only on 1=0;

5.       Using the Unity Director Dictionary Scanner, create a dictionary for the database. Be sure to include the system2Only table, and historyView on system 2. Since the historyView depends on the system2Only table, it will automatically be available only on system 2. This is important, because Unity Director 14.00 will not otherwise allow the view to be selectively managed on a specific system.

6.       Historical data (records for 10 years in this case) should be populated directly on system 2:

insert into basedata values (1, CAST('20030503111111' AS TIMESTAMP(6) FORMAT 'YYYYMMDDhhmiss'));
insert into basedata values (2, CAST('20040503111111' AS TIMESTAMP(6) FORMAT 'YYYYMMDDhhmiss'));
insert into basedata values (3, CAST('20050503111111' AS TIMESTAMP(6) FORMAT 'YYYYMMDDhhmiss'));
insert into basedata values (4, CAST('20060503111111' AS TIMESTAMP(6) FORMAT 'YYYYMMDDhhmiss'));
insert into basedata values (5, CAST('20070503111111' AS TIMESTAMP(6) FORMAT 'YYYYMMDDhhmiss'));
insert into basedata values (6, CAST('20080503111111' AS TIMESTAMP(6) FORMAT 'YYYYMMDDhhmiss'));
insert into basedata values (7, CAST('20090503111111' AS TIMESTAMP(6) FORMAT 'YYYYMMDDhhmiss'));
insert into basedata values (8, CAST('2010503111111' AS TIMESTAMP(6) FORMAT 'YYYYMMDDhhmiss'));
insert into basedata values (9, CAST('20110503111111' AS TIMESTAMP(6) FORMAT 'YYYYMMDDhhmiss'));
insert into basedata values (10, CAST('20120503111111' AS TIMESTAMP(6) FORMAT 'YYYYMMDDhhmiss'));


select count(*) from basedata;

*** Query completed. One row found. One column returned.
*** Total elapsed time was 1 second.

   Count(*)
-----------
         10

7.       The data for the current year can now be populated via Unity Director,using the currentYearView:

insert into currentYearView (11, CAST('20130503111111' AS TIMESTAMP(6) FORMAT 'YYYYMMDDhhmiss'));
 *** Insert completed. One row added.
 *** Total elapsed time was 1 second.


select count(*) from currentYearView;
 *** Query completed. One row found. One column returned.
 *** Total elapsed time was 1 second.


   Count(*)
-----------
          1

8.       Selects to the history view will automatically route to System 2, and see the full data, because the system2only is only present on system 2:

select * from historyView order by ts;

*** Query completed. 10 rows found. 2 columns returned.
*** Total elapsed time was 1 second.

         id                          ts
-----------  --------------------------
          1  2003-05-03 11:11:11.000000
          2  2004-05-03 11:11:11.000000
          3  2005-05-03 11:11:11.000000
          4  2006-05-03 11:11:11.000000
          5  2007-05-03 11:11:11.000000
          6  2008-05-03 11:11:11.000000
          7  2009-05-03 11:11:11.000000
          9  2011-05-03 11:11:11.000000
         10  2012-05-03 11:11:11.000000
         11  2013-05-03 11:11:11.000000

Daily Modifying and Loading/Unloading of Current Data

Inserts of new current data should be done through Unity Director using the current data view (currentYearView) as shown in step 6 of the previous example. Unity Director will multicast these inserts to both Teradata systems.

Using the current data view reduces the risk that a user will unintentionally modify historical data that exists only on the second system, since the historical data is not available via this view:

delete currentYearView where id=2;

 *** Delete completed. No rows removed.
 *** Total elapsed time was 1 second.

Updates or deletes of current data can also be executed via Unity Director, similarly using the current data view to prevent touching historical data.

delete currentYearView where id=11;

 *** Delete completed. One row removed.
 *** Total elapsed time was 1 second.

In the event a user attempts to modify historical records through Unity Director that only exist in the second system, Unity Director will detect this as a data inconsistency and make one of the copies of the table offline.

Consider this ill-advised update to the base data table from a rogue user, accessing the basedata table directly:

update basedata set id=id;

 *** Update completed. One row changed.
 *** Total elapsed time was 1 second.

Since this update to the base data table affected 1 row on system 1, and 10 rows on system two, Unity Director detects the data inconsistency and takes one copy of the table offline, generating the following alert (some text removed for brevity - note the differing row counts):

Alert No.          : 99
Alert Code         : 40054
Alert Description: An inconsistent response was detected from a Teradata system.  As a result the object has been placed in the Unrecoverable state.
Alert Details      : System 2 (db2) sent an inconsistent response, table dbtest.basedata is unrecoverable.

Response mismatch: session 1000, txn 376, request 13, log 11421536
…
        SQL: 'update basedata set id=id;'…
Response 1 from system 1 (db1): row count 1, hash 3010, mesh status 0, DRCP code 0 [sent to client]
…
Response 2 from system 2 (db2): row count 10, hash 30a0, mesh status 0, DRCP code 0
…
System 2 (db2) sent an inconsistent response, table dbtest.basedata is unrecoverable.

Which copy of the table (system 1 with only current data, or system 2 with the historical data) is taken off-line is not-deterministic. Since Unity Director is designed to protect the user against these data inconsistencies, the rogue user sees only 1 row affected, since system 1 was the first to respond in this case.

Unloading, Archiving or Modifying Historical Data

Since the historical data cannot be modified via Unity Director or Loader, ETL processes that modify that data must access the data directly on the second system. As with any process directly accessing a Unity Director managed table on a single Teradata system, these direct ETL processes must be managed appropriately to not interfere with any work load accessing the base table via Unity Director or Unity Loader. The base table should be halted in Unity Director before attempting to load or unload any historical data, and then recovered to return it to active service when the ETL process is complete.

unityadmin> object halt dbtest.basedata on db2;

The request is currently processing as operation number 25.
You may check its status using the command 'operation check 25'.

unityadmin> operation check 25;

Operation Number : 25
Operation Name   : Halting Table
User             : admin
User Name        : Main Administration User
Progress (%)     : 100
Status           : Finished (1)
Start Time       : 05/14 07:13:17
Finish Time      : 05/14 07:13:18
Systems:

    [2] db2 - Finished (1)

Updates:

    05/14 07:13:18 [-] Info: Halting table dbtest.basedata
    05/14 07:13:18 [-] Info: Requesting mgmt X lock on 'dbtest.basedata'
    05/14 07:13:18 [-] Info: Mgmt X lock on 'dbtest.basedata' granted
    05/14 07:13:18 [-] Info: Releasing mgmt X lock on 'dbtest.basedata'
    05/14 07:13:18 [-] Info: Successfully halted table on 1 systems
    05/14 07:13:18 [-] Info: Operation finished

unityadmin>

It's important to note that during this operation, the historical data will be unavailable to client applications via Unity Director. Clients attempting to read from the historical view will automatically hold until the table is returned to service.

BTEQ -- Enter your SQL request or BTEQ command:

select * from historyView order by ts;

…client application waits until the base table is returned to service, since it requires both the system2Only table, and the basedata table.

Once the base table is recovered:

unityadmin> object recover  dbtest.basedata on db2;

The request is currently processing as operation number 26.

…then the client request completes:

select * from historyView order by ts;

 *** Query completed. 10 rows found. 2 columns returned.
 *** Total elapsed time was 3 minutes and 49 seconds.

         id                          ts
-----------  --------------------------
          1  2003-05-03 14:11:11.000000
          2  2004-05-03 14:11:11.000000
          3  2005-05-03 14:11:11.000000
          4  2006-05-03 14:11:11.000000
          5  2007-05-03 14:11:11.000000
          6  2008-05-03 14:11:11.000000
          7  2009-05-03 14:11:11.000000
          9  2011-05-03 14:11:11.000000
         10  2012-05-03 14:11:11.000000
         11  2013-05-03 14:11:11.000000

 BTEQ -- Enter your SQL request or BTEQ command:

Availability Considerations

As shown in the previous example, if the base data table is unavailable for any reason on the second system, the historical data will be unavailable. If the table is out-of-service or standby, any reads against the historyView will hold until it is returned to server.

Should the base data table become unrecoverable on the second system, then any queries against the historyView will fail with the following error:

select * from historyView order by ts;

 *** Failure 4510 No systems available
 *** Total elapsed time was 1 second.

Unity Director will also generate an alert when this occurs:

Future Changes in Unity Director 14.10

The use of this system2Only table is only a temporary work around, necessary for the 14.00 version of Unity Director. Unity Director 14.10 will introduce the ability to select which systems views are managed on, eliminating the need for this table. This will eliminate the need to add the locking modifier for the historyView:

create view historyView as
 locking basedata for access
 select * from basedata

Instead, the Unity Director's Data Dictionary will explicitly allow the view to be managed on specific systems, without the dependency on the system2Only table:

Additional Resources

• Unity Director 14.00 & Unity Loader 14.00 release announcement
• Unity Director 14.00 - Routing Rules
• Unity Director User Guide
• Unity Director brochure
• MicroStrategy reporting using Teradata Unity Director 14.00
• Unity Loader brochure
• For community support, please visit the Ecosystem forum.