Search - User list
Full Version: Attribute Merging with Partitioners
Root » PDG/TOPs » Attribute Merging with Partitioners
bsilvaLS
Hello!

When you partition, it merges the lists of Files in the pdg_output attribute. I was hoping it would do the same with string attributes, since they're inherently arrays, but this doesn't seem to be the case.

Is there a way to do this, short of writing a Python partitioner? Or can someone recommend an alternate workflow?

Thanks much!
tpetrick
The input and output file are special attribute types that behave differently than other attributes on work items. They're internal to PDG and maintained automatically by the graph, which includes passing upstream outputs -> work item inputs, and merging them in partitioners. The merging functionality is currently not available for other types of attributes when partitioning, although that is an RFE we plan to tackle.

Even in a custom partitioner, it's not possible for user-defined code to edit the attributes on a partition. The partitioner copies attributes onto itself automatically based on the sort order of work items in the the partition (see the Advanced tab for those parms). This happens at different points in the cook depending on if the partition is static or dynamic. That makes it somewhat more challenging to provide a consistent implementation, but again it is a feature that we intend to add eventually.

In the mean time, the best solution is to use the partitioner you want, and then follow it with a Python Script that extracts the attributes you need. Your Python Script has access to the parent item and if that's a partition you can access the items in the partition:

merged = []
for partition_item in parent_item.partitionItems:
    merged += partition_item['str'].values
work_item.setStringAttrib("merged", merged)

You'll want to be on the build that lets you run the Python Script code in the generate, as mentioned earlier in a different post. If you're not you can still use a Python Processor, but you'll have to manually construct new work items (which is what that node is for):

for upstream_item in upstream_items:
    merged = []
    for partition_item in partition_items:
        merged += partition_item['str'].values
    new_item = item_holder.addWorkItem(parent=partition_item)
    new_item.setStringAttrib("merged", merged)
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.
Powered by DjangoBB