As shown below, the “x2” notation indicates that two copies will be started at runtime. The platform delivers accurate.To take advantage of the processing resources in your server, you can scale up the transformation using the multi-threading option ‘Change Number of Copies to Start…’ to produce copies of the steps (right click the step to access the menu). Please note there may be issues with other values other than integer.Pentaho Data Integration prepares and blends data to create a complete picture of your business that drives actionable insights. -integer : scaling will be in integer multiples-exact : scaling factor will use exact scaling factor set in the system settings-quarter : same as exact but scaling factor is rounded off to nearest 25- : you can specify a scaling factor value which you want to use.

Reading the data from the CSV file is done in parallel. Row N+1 goes to the first copy again, and so on until there are no more rows to distribute. This means that if there are N copies, the first copy gets the first row, the second copy gets the second row, and the Nth copy receives the Nth row.

The names used for the partitions can be anything you like. A partition schema defines how many ways the row stream will be split. First, configure a partition schema. The idea is to establish a criterion by which to partition the data, so that resulting storage and processing groups are logically independent from each other. Now, the ‘count by state’ aggregation step produces consistent correct results because the rows were split up according to the partition schema and rule, as shown in the preview data.Note: To view this transformation in the PDI client, open the Pentaho/…/design-tools/data-integration/samples/transformations/General - parallel reading and aggregation.ktr sample file.The partitioning method you use can be based on any criteria, can include no rule (round-robin row distribution), or can be created using a partitioning method plugin. In the example below, a partition schema called 'State' was applied to the ‘count by state’ step and the ‘Remainder of division’ partitioning rule was applied to the ‘state’ field.

If the modulo is calculated on a non-integer value, the PDI Client calculates the modulo on a checksum created from the String, Date, and Number value.When you run the transformation, there are no guarantees as to which page name goes to which step copy, only that any page name encountered is consistently forwarded to the same step copy.Partitioning data allows your transformations to scale out on a cluster of slave servers to maximize the resources of machines operating in parallel. The ‘Remainder of division’ rule allows rows with the same state value to be sent to the same step copy and the distribution of similar rows among the steps. Establish the partitioning method for the step, which defines the rule for row distribution across the copies. By applying a partition schema to a step, a matching set of step copies is started automatically (for example, if applying a partition schema with three partitions, three step copies are launched).Step Two, select the partitioning method:

Also, to get optimal performance on a cluster, try to keep the data in swimlanes, described above, for as long as possible. It is, therefore, recommended to allow the PDI client to create the partition schema dynamically in a clustered environment.You should always limit repartitioning on a cluster to a minimum, as high amounts of networking and CPU overhead can be incurred, which is caused by the massive amounts of data passing from one server to another over TCP/IP sockets. As a result, the number of partitions needs to be equal to or higher than the number of slave servers in the cluster schema.

Pentaho Data Integration Scaling