I'm starting this thread as a result of a separate discussion here [sidefx.com].
My intent is to give a rough sketch of a VOP packaging system – a tool that can be used to build VOPs straight from text sources. This is a desirable thing when designing a build system for all things VEX while also supporting a version-control system in a “natural” way.
I may end up posting the whole finished thing in the forum some day, and if I do, this will serve as a road map for how all the bits fit together, so hopefully this is not a redundant effort… (I actually copy/pasted most of the stuff here from my packager's help, so I didn't have to type much)
REQUIREMENTS:
The overriding requirement is this: we wish to be able to write a “vop” file which when “compiled” results in an OTL that defines *every* *single* *aspect* of the VOP – no “dangling” bits allowed
Furthermore, we wish to be able to take *full* advantage of the usual development tools: macros, includes, etc. In short, we want to be able to write these things in much the same way we write VEX or C++ code, and all that that implies. This, in turn, will address our other important requirement, which is that we want to keep all VOP sources under CVS control in the same way we keep all other sources – since they're written in the same way, it should follow that they can be maintained in the same way.
Now to the specifics….
VOPs are just a dialog script with “code”, “outer-code”, “input”, “output”, and “signature” sections – a dialog script on steroids . That's it. All other adornments, such as Icon images, callback scripts, related DSOs, etc; are the domain of the OTL that wraps them – they have to be dealt with as well, of course, but they don't actually *define* the VOP: they are just resources used by the VOP.
So we'll leave those for later and concentrate on this dialog script thingy first
The dialog script is made up of different sections that represent different “kinds” of things (this is actually part of a bigger problem, since they are themselves a kind of OTL, but that's a separate thread ). It should come as no surprise then, that each section carries its own syntax. For VOPs, our packager needs to deal with the following sections:
ID:
Things that identify the OP. Some are defined in the dialog script, and some in the OTL. These include:
Name
Label
Max/Min Inputs
Icon (optional)
NetMask (optional)
Help:
A block that begins with "help {“ and ends with a closing ”}“. Each line in the block is quoted, and all embedded quotes are escaped with a backslash ”\".
Parameters:
Parameters are not contained within a block – they live in the main “body” of the script and follow their own syntax rules. The general syntax for a single parameter is:
parm {
name
label
……
}
Note that the following is equally legal (though much harder to read)
parm { name var1 label “My Input Var” … }
…meaning that macro expansions (which end up as a single line) can be exploited to great advantage
Parameters can live in the main client space of the UI, or be assembled into groups. Each group will become a “tab” in the final UI. Groups can be embedded within groups, ending up with tabs within tabs. A group block has the same general syntax as a parameter block, but (to my knowledge) only accepts two (required) token-value pairs. These are (like for parm{}) a “name” and a “label”. The label field is what gets displayed as the tab's name, and the name field is presumably used internally to id the group – although I suspect it's a placebo field at the mo. The keyword that starts a group block is, appropriately: “group”
Here then, is an example with one floating parameter, followed by a tabbed group with two parameters:
parm {
name var1
label “Anti-Social”
…
}
group {
name g1
label “Hello World”
parm {
name var2
label “Groupie I”
…
}
parm {
name var3
label “Groupie II”
…
}
}
Again; it is perfectly legal to write all that in a single line. Can you say “macro”?!
I've been writing ellipses (…) for the implicit guts of each parameter in the examples so far. What goes in there is really more token-value pairs giving more details about each parameter. Here's a complete list of (to my knowledge) all the allowable token value pairs and their corresponding meanings:
name
label
type
<opfilter>
size
default val
disable { {token value} {token value} … }
callback
range { lo_val hi_val }
export
invisible
menu { “tok” “val” “tok” “val” … }
The type is really either int, float or string. But with all the hints,
they expand to:
integer
float
vector
vector4
uv
uvw
string
toggle
button
angle
file
image
geometry
oppath
direction
color
color4
Inputs:
The “inputs” section describes the name and type of inputs expected by this VOP. A VOP can have more inputs than parameters or more parameters than inputs, it doesn't matter – although they are usually somewhat synchronized.
However, if your VOP has 100 parameters, you probably won't want to expose each and every one as an input!
The packager should take care of counting the number of inputs and setting that as the “max inputs” field in the OTL for you.
The syntax for each input line is quite simple:
input
Where “vex_type” is an actual vex type: int, float, vector, vector4, or string (I haven't tried the matrix types yet). “parm_name” is the variable name for that parameter/input and “label” in this case is the string that gets displayed when you hold the mouse over the widget – the “tooltip”.
Here are some examples:
input vector pos Position
input float oct Octaves
input float fsize “Filter Size”
Outputs
As you'd imagine, this section describes the number and types of outputs that this VOP will produce. Our packager should count them and set the “max output” field in the OTL for you.
Much like inputs, the syntax for these is very simple:
input
And they carry the same meaning as with the inputs.
Signatures
Signatures describe variations in the allowed types for either the inputs or the outputs (or both) for a VOP. The so-called “default” signature, is simply the input/output types as defined within the dialog script (see the Input and Output sections above). But you are also allowed to specify alternative types for one or more of these inputs/outputs; each alternative set then becomes a different “signature” for your VOP (function).
The syntax for each signature is:
signature { input_types output_types }
Please note that “name” and “label” are in reverse order from most of the other dialog commands. Watch out!
The block of types is a space-separated list of the vex-type associated with each input and output respectively. Their order should match the order in which they were defined in their respective sections.
Each VOP must have at least one signature, its name must be “default”, it must have an empty type block, but it can have any descriptive label.
IMPLEMENTATION:
First, let's arbitrarily say that VOP-type files will have the extension “.vop” – in case this can be used as a discriminant somewhere in our system, to differentiate them from, say, “.h” or “.vfl” or “.C” files.
One of our main requirements is that we be allowed to use “standard” VEX/C/C++ syntax, along with #include directives, macros, etc – the whole deal. The solution that immediately suggests itself is to do a C-Preprocessor pass as a first step in our process; this will do all the things we want, as well as something we don't want: it will, by default, pollute the result with all kinds of “File/LineNumber” entries. A quick look at the man pages for the cpp (c-preprocessor) that's bundled with Red Hat reveals that the option “-P” suppresses line numbers from the output (your version may be different). Great; full C++ syntax; done!
We'll then need a second pass that will interpret the contents of our file, and format/assemble them in a way that's recognizable to Houdini as a “dialog script”. The classical way to do this is to embed symbols that are only meaningful to our interpreter, but otherwise transparent to the cpp pass, and which don't show up in the final result. I've arbitrarily chosen the character '@' to signal the beginning of a keyword for my interpreter – lots of other equally successful possibilities here.
So; keywords for our interpreter will be formed by the character '@', followed by something meaningful. Let's also say that these keywords will only occur at the beginnings of lines; no other location is allowed – this just simplifies things
Given that some sections need begin/end code (such as the “Help” section), we'll need to formalize that syntax a bit. Let's tentatively say that our begin/end blocks will be defined by "@<token> BEGIN“ and ”@<token> END" pairs. Additionally, we'll say that if one exists, then so should the other – anything else represents a syntax error.
Armed with these specs, we can sketch what our vop syntax will look like, here's an early attempt:
@NAME myNoise
@LABEL “My Noise”
@ICON /some/image/file.pic
@CONTEXT “surface cop2filter cop2gen sop”
#include <useful1.h>
#include <useful2.h>
@HELP BEGIN
// This should be stripped by cpp, thank-you-very-much
This is the help for my vop… dig it!
@HELP END
parm {
name pos
label “Position”
type vector
}
input vector pos Position
output vector chaos Noise
signature default {}
@CODE BEGIN
vector pp = $is_connected_P ? pos : P;
$chaos = myOuterFunction($pp);
@CODE END
@OUTER BEGIN
vector myOuterFunction(vector p) {
return vector(noise(p));
}
@OUTER END
The amount of encoding going on is so minimal that even a passing familiarity with any scripting language (awk, perl, python, tcl, whatever) should be sufficient to write an interpreter for that!
So; here's the emerging picture of our packager:
- 1. It's a shell script that takes one or more “.vop” files as input.
2. For each file it:
A) Runs cpp on it. The output of this step gets passed to…
B) A custom interpreter which looks for ‘@’ tokens and formats them appropriately. Tokens that it doesn't understand (if any) are left alone so they can be interpreted by the caller. I ended up writing mine in about 40 lines of awk (haven'thad time to learn perl yet), and embedded them inside the shell script so the whole thing is one file – another pretty standard trick with unix shell scripts
3. Using “hotl”, the shell script creates a stub OTL and inserts the result of step #2 as the dialog script, and then adds other sections as indicated by the remaining commands embedded (through the same ‘@’ mechanism) in the .vop file that weren't consumed by step #2: things like adding an icon file, callback scripts, whatever – things that need to happen at the OTL-level, as opposed to the dialog-script level.
Looking at our vop fragment above, you'll no doubt note that the input, output, and signature sections needed no special treatment, so they were not accompanied by a ‘@’ keyword.
Also; alarm bells should be going off when looking at the parameter definition – these things could get mighty lengthy for an actually useful VOP! Not only that; I need to learn a whole new syntax for defining parameters, yikes!
Not to worry… we're in luck:)
Remember we're doing a cpp pass… this means full macro expansions are allowed! (also recall that all these things can be defined in a single line – which is what macros expand to).
Say we define a header like this:
file: vop_stuff.h
#ifndef vop_stuff_h_GUARD
#define vop_stuff_h_GUARD
#define VOP_PARM(TYPE,NAME,LABEL,DFLT,RNG,DSBL) \
parm { \
name NAME \
label LABEL \
type TYPE \
default { DFLT } \
}
#endif
Now I can define the parameter for my vop as follows:
…
#include <vop_stuff.h>
…
VOP_PARM(vector,P,Position,0 0 0)
…
One line, and now it sports a default value… that's *much* better
Any programmers out there can, I'm sure, see what kind of automated “yummyness” this can lead to
Same goes for the other sections – for example, in my current version, I can define an HTML table and a row entry with custom (embedded) style sheets in 4 lines of code (code that doesn't interfere with the main .vop file because it's included through #include<>)… you get the picture.
Obviously; once everything is working and all the basic macros are written, this turns into a pretty slick and efficient way to write VOPs. Also; given that everything is now text-based source files, we can keep vop definitions in the same development structure as our VEX shaders and our C++ DSOs… and put the whole thing under CVS. Sweet!
hmmmm…. I think I'll stop there for now. This should give an idea of where to start at least – it *looks* a lot more complicated than it really is; trust me!