Tonight I am finishing up work with interfacing the Houdini python interface, i.e. the HOM, to AFX and Mathematica. This enables AFX to increase its capabilities by having seamless access to all the Houdini python work that SideFx has done. Performance wise the HOM is slow but it was straightforward to integrate into AFX, so we did. An important point is that the size of the Houdini networks created by AFX will be much larger than is typically possible using Houdini or scripting so it is important to wrangle the networks in AFX so that when the networks are seen and manipulated in Houdini they are not a giant flat sea of nodes as in the above screen shot.

With AFX you can use Houdini to do everything and more than MASSIVE provides and much easier as well. MASSIVE makes heavy use of fuzzy logic, which has fundamental limitations. Fuzzy logic, for those not familiar with the technology, is simply sets of things that have approximate membership. An example is the following non fuzzy set:

A set of arbitrarily chosen dogs lets say for example: {pitbull, poodle, collie}. If I wanted to include wolf in this set that would be invalid however with fuzzy logic sets can have approximate membership, so in our example (pitbull 100%, poodle 100%, collie 100%, wolf 75%} is a valid fuzzy set. Fuzzy logic set operations include ways to manipulate the probabilities. The problem with fuzzy logic is that the probabilities are established by whoever created the fuzzy logic set (in our example). This is a fundamental and non-scalable flaw with regard to fuzzy logic.

Over the decades since Zadeh introduced fuzzy logic there have been many attempts to automate establishing the probabilities inherent to fuzzy logic but its ad-hoc.

A better way is to use self-learning artificial neural networks (ANN), which are recurrent and connectionless. Connectionless ANN can be thought of as ANN nodes that broadcast what they are doing on different frequencies (think of a cell phone network). In a connectionless ANN the network topology is an emergent property of these interactions. This addresses all current challenges with ANN. We are implementing connectionless ANN in AFX and Houdini will have new nodes to exploit it. The results should be far superior to MASSIVE technology. The problem with MASSIVE is that the output of that technology has the same look and feel due to the fuzzy logic, which predominates the technology, so all MASSIVE simulations sort of look and feel the same. Using connectionless ANN the look and feel evolves with the simulation as the ANN's learn. An example of connectionless ANN solving a difficult problem is here : http://fvg.com/heat-seeker-algorithm.html [fvg.com] Just finished testing AFX HOM integration and it works great.

COPs + Trace SOP ?

blackpixel
COPs + Trace SOP ?

Interesting! did you do this with COPS and Trace SOP? How how complex is the COPS? Thanks for posting this. When I used the trace I obtained the same basic outline of your result around the face and the eyes. I realized that a cops was necessary but I wanted to finish the animation of the model first before I went back to work on the COPS since I have not done that in Houdini before. How complex was the COPS to implement? This is very helpful, thanks so much for posting.

VR

Lars

edward
Is that 5657 object nodes for representing each NURBS curve?

PS. I wonder how different the Trace SOP [sidefx.com] result would compare.

Edward, sorry I missed the first part of your question. There are 5657 curves total each with no more than 20 control points, however it is possible to isolate different parts of the face in AFX/Mathematica to have higher definition curve generation and then combine the resulting curves either as separate clusters of curves in Houdini or individual curves as I did here. Then merge the curves (or use the group shelf tool) and you can start to use Houdini as usual, at least that is how I am proceeding at this point. Except for the soundtrack lipsync, which I am processing with AFX Mathematica codes to extract the voice and phonemes as input to a Houdini CHOP network. Then I will write the code in AFX to do the animation and see how they compare. Also note that Mariusz has done the sop + cop as you requested above for comparison. I am very interested to see what he did on the cop. Perhaps he created a mask by desaturating the image as input to the cop or maybe used RGB values in the cop, I am very curious to know.

I just used some filters, color correction and the chroma keyer to get a better base image for the Sobel edge detect.

blackpixel
I just used some filters, color correction and the chroma keyer to get a better base image for the Sobel edge detect.

Again thanks so much!

Currently animating the model. It deforms easily since it is curves based. I created the background content and audio track for the animation. I am using AFX Mathematica code to do most of the animation algorithmically, which translates directly to Houdini. The idea here is to have this be a completely automated process using AFX/Houdini virtual machine. Once I have the bugs worked out of the 2D process I will move to doing this in 3D and go through through the same to optimize the 3D process. I move back and forth from Houdini to see the results and see how AFX can better use Houdini as virtual machine. The 2D AFX video model is modified with the its animation controls so the chin and mouth are closer to the rest position.

http://fvg.com/the-daily.html [fvg.com]

FVGDOTCOM
http://fvg.com/the-daily.html [fvg.com]

Is that video there supposed to have the 2D model deforming over time in some fashion? I watched the full video but didn't notice the curves changing at all.

that is actually a really nice trace operation on the image great to see that a lot of the noise that you would normally see on the skin is actually gone (and for nurbs, you do not see a lot of “nurbs explosion…!”

It may actually be more efficient to just combine all curves into a single SOP node VS having multiple objects

in regards to what is Art, I think the best definition that I ever heard (with the caveat that I cannot remember the source for it), is that Art is:

“ The arrangement of any elements in such a way that it entices a feeling or emotion”

nice and simple

elements can be anything (lines, shapes, audio, video, geometry, points flying around, etc), and the result is simply a feeling or emotion (pleasure, dislike, excitement, disgust, etc). in this particular definition, a mathematical algorithm which is able to entice some feeling or emotion could be defined as such (think about the use of geometrical concepts such as the golden ratio in renaissance art).

BTW it would be interesting to see how your algorithm fares against video. it would be cool to see if you can use the first frame for registration, and try to maintain the nurbs topology as intact as possible in between subsequent frames

That sounds like the new Vector Distort tools in Nuke 10.It is excellent at tracking and then distorting from a reference frame.

https://www.youtube.com/watch?v=65QkIydYMSQ [youtube.com]

Artye
That sounds like the new Vector Distort tools in Nuke 10.It is excellent at tracking and then distorting from a reference frame.

https://www.youtube.com/watch?v=65QkIydYMSQ [youtube.com]

Cool! Thanks for the link, I always interested in other technologies.

grayOlorin
BTW it would be interesting to see how your algorithm fares against video. it would be cool to see if you can use the first frame for registration, and try to maintain the nurbs topology as intact as possible in between subsequent frames

I have experimented recently with video frames extracted using a photoshop frame render and AFX works great against video frames. Before I extend the algorithms to 3D I am going to use video frames to automatically generate blend shapes, which I am doing algorithmically this time but I see how reduced workflow computational time and increased morph target accuracy will result using curve video motion capture.

grayOlorin
that is actually a really nice trace operation on the image great to see that a lot of the noise that you would normally see on the skin is actually gone (and for nurbs, you do not see a lot of “nurbs explosion…!”

It may actually be more efficient to just combine all curves into a single SOP node VS having multiple objects

in regards to what is Art, I think the best definition that I ever heard (with the caveat that I cannot remember the source for it), is that Art is:

“ The arrangement of any elements in such a way that it entices a feeling or emotion”

nice and simple

elements can be anything (lines, shapes, audio, video, geometry, points flying around, etc), and the result is simply a feeling or emotion (pleasure, dislike, excitement, disgust, etc). in this particular definition, a mathematical algorithm which is able to entice some feeling or emotion could be defined as such (think about the use of geometrical concepts such as the golden ratio in renaissance art).

Thanks so much!

edward

FVGDOTCOM
http://fvg.com/the-daily.html [fvg.com]

Is that video there supposed to have the 2D model deforming over time in some fashion? I watched the full video but didn't notice the curves changing at all.

That is what I am working on now. I am fine tuning the algorithm and the AFX/Houdini interactions to optimize performance. Particularly working with blendshapes automation. Its turning out that AFX/Houdini is perfect combination for this. Much better than using Maya, C4D or 3ds, by orders of magnitude in terms of workflow efficiency. Using CHOPS and COPS fits great into the workflow.

Houdini also fits naturally with my algorithmic proficiencies. AFX amplifies Houdini capabilities, it is not a replacement for Houdini. Houdini is the engine and AFX is its supercharger.

2D model ready to be deformed. This workflow took one minute from image input to Mantra render. Detail is easily adjusted or changed using multiple reference images, including compositing of curves. In this example I intentionally filtered the image to make the right side of the face look normal and the left side the joker, two sides of the same coin. Any image, groups of images, video or groups of video can be used as input. Morph targets connected to Houdini edit nodes terminating at a blendshape node are automatically generated from video input. I am working to have AFX wire and integrate Houdini phoneme and voice sync nodes to the morph targets from a noisy audio speech source.

More information on this post: http://fvg.com/news.html [fvg.com]

Completed writing AFX algorithms to perform automatic audio analysis, which precisely separates background noise and music from the artist's voice, including re-constructing the speech of the voice. This algorithm uses a Wavelet analysis filter I invented for this process. Including new Houdini CHOP nodes, which automate for perfect lip sync. Below is the Wavelet analysis plot converted to SI Hertz units. This algorithm works better than I developed to detect pesky Soviet clandestine Nuclear Tests. http://fvg.com/cold-war.html [fvg.com]

More information on this validation study here: http://fvg.com/validation.html [fvg.com]

Here is the first pass at the animation. There is no key framing. Everything is AFX and voice CHOPS based, completely automatic driving the morph targets from audio. Change the animation and lip sync simply change the audio track. New AFX NURBS curve models reuse the same AFX/CHOPS network with new audio track, which is automatically processed by AFX.

http://fvg.com/the-daily.html [fvg.com]

grayOlorin
in regards to what is Art, I think the best definition that I ever heard (with the caveat that I cannot remember the source for it), is that Art is:

“ The arrangement of any elements in such a way that it entices a feeling or emotion”

nice and simple

elements can be anything (lines, shapes, audio, video, geometry, points flying around, etc), and the result is simply a feeling or emotion (pleasure, dislike, excitement, disgust, etc). in this particular definition, a mathematical algorithm which is able to entice some feeling or emotion could be defined as such (think about the use of geometrical concepts such as the golden ratio in renaissance art).

What art is would be a quite complicated discussion.
What it is not, might be simpler.
But to be art, something needs to have intent. No, the simple default output of some algorithm, even it might be “nice” to look at, is not art. The same as a superb landscape, or a rainbow after a storm, even if they might fill you with quite powerful emotions, are not art.

I agree with that. I think in that definition, arrangement of elements implies that there was some “intention” on taking elements (from wherever they come) and arranging them in a certain fashion (meaning not necessarily how they are arrange by nature. Nature phenomena does not indeed count as art)

Indeed, the default output of an algorithm is not art on itself, but using an algorithm as a means to arrange elements could be (i.e the golden ratio is not art, but paintings using the golden ratio to give a particular feeling is)