Hello!
When you partition, it merges the lists of Files in the pdg_output attribute. I was hoping it would do the same with string attributes, since they're inherently arrays, but this doesn't seem to be the case.
Is there a way to do this, short of writing a Python partitioner? Or can someone recommend an alternate workflow?
Thanks much!
Attribute Merging with Partitioners
1631 1 2-
- bsilvaLS
- Member
- 11 posts
- Joined: April 2019
- Offline
-
- tpetrick
- Staff
- 616 posts
- Joined: May 2014
- Offline
The input and output file are special attribute types that behave differently than other attributes on work items. They're internal to PDG and maintained automatically by the graph, which includes passing upstream outputs -> work item inputs, and merging them in partitioners. The merging functionality is currently not available for other types of attributes when partitioning, although that is an RFE we plan to tackle.
Even in a custom partitioner, it's not possible for user-defined code to edit the attributes on a partition. The partitioner copies attributes onto itself automatically based on the sort order of work items in the the partition (see the Advanced tab for those parms). This happens at different points in the cook depending on if the partition is static or dynamic. That makes it somewhat more challenging to provide a consistent implementation, but again it is a feature that we intend to add eventually.
In the mean time, the best solution is to use the partitioner you want, and then follow it with a Python Script that extracts the attributes you need. Your Python Script has access to the parent item and if that's a partition you can access the items in the partition:
You'll want to be on the build that lets you run the Python Script code in the generate, as mentioned earlier in a different post. If you're not you can still use a Python Processor, but you'll have to manually construct new work items (which is what that node is for):
Even in a custom partitioner, it's not possible for user-defined code to edit the attributes on a partition. The partitioner copies attributes onto itself automatically based on the sort order of work items in the the partition (see the Advanced tab for those parms). This happens at different points in the cook depending on if the partition is static or dynamic. That makes it somewhat more challenging to provide a consistent implementation, but again it is a feature that we intend to add eventually.
In the mean time, the best solution is to use the partitioner you want, and then follow it with a Python Script that extracts the attributes you need. Your Python Script has access to the parent item and if that's a partition you can access the items in the partition:
merged = [] for partition_item in parent_item.partitionItems: merged += partition_item['str'].values work_item.setStringAttrib("merged", merged)
You'll want to be on the build that lets you run the Python Script code in the generate, as mentioned earlier in a different post. If you're not you can still use a Python Processor, but you'll have to manually construct new work items (which is what that node is for):
for upstream_item in upstream_items: merged = [] for partition_item in partition_items: merged += partition_item['str'].values new_item = item_holder.addWorkItem(parent=partition_item) new_item.setStringAttrib("merged", merged)
Edited by tpetrick - April 10, 2020 13:08:14
-
- Quick Links