... | ... | @@ -7,11 +7,9 @@ Myriad Data Generator Toolkit |
|
|
Core Features
|
|
|
-------------
|
|
|
|
|
|
The *Myriad Toolkit* has two main components: a generic C++ *runtime library* for scalable data generation, and a Python *prototype compiler* that generates library extensions from a user-defined data generator specification written in XML.
|
|
|
The *Myriad Toolkit* has two main components: a generic C++ *runtime library* for scalable data generation, and a Python *prototype compiler* that generates library extensions from a user-defined prototype specification data generator written in XML.
|
|
|
|
|
|
An XML specification defines the structure of the generated *domain model* as a family of user-defined *domain types*, and the data generation logic as a corresponding family of *pseudo-random domain type generators (PRDGs)*.
|
|
|
|
|
|
Essentially, a PRDG is a function that generates a sequence of pseudo-random domain type records from an underlying sequence of pseudo-random numbers. In the XML specification PRDGs are realized as a chain of *setter functions*. Applying a setter to a generated record assigns (i.e. sets) a specific value to one or more of its components. The *Myriad Toolkit* provides a range of primitive setters that implement various statistical properties (e.g. value distributions in a record fields or value dependencies between several record fields).
|
|
|
The XML specification contains the structure of the generated *domain model* as a family of user-defined *domain types*, and the data generation logic as a corresponding family of *pseudo-random domain type generators (PRDGs)* - functions that generate a sequence of pseudo-random domain records from an underlying sequence of pseudo-random numbers. PRDGs are realized as chains of *setter functions*. Applying a setter to a generated record assigns (i.e. sets) a specific value to one or more of its components. The *Myriad Toolkit* provides a range of primitive setters that implement various statistical properties (e.g. value distributions in a record fields or value dependencies between several record fields).
|
|
|
|
|
|
Besides the simple specification language, the *Myriad runtime library* transparently builds-in parallelization support in all compiled data generators. To do so, the framework makes sure that the following two conditions always hold. First, each domain record is identified by a unique position (i.e. a concrete seed) in the generating pseudo-random number sequence. Second, the sequence of pseudo-random numbers is generated by a pseudo-random number generator (PRNG) function that supports arbitrary skips to any position on the sequence in constant time. These runtime-level decisions are critical for efficient parallelization, as they allow us to (A) partition the generated PRDG sequences across arbitrary number of data generator nodes in a shared-nothing environment, and (B) use *function shipping* (i.e. re-compute) instead of *data shipping* (i.e. transfer over the network) to get the contents of a referenced record generated on a remote node.
|
|
|
|
... | ... | |