I want the color to only effect the concave areas of the mesh, instead the other areas of the mesh are being effected as well ?

You're using the dot product between the normal and the Y axis, so this will select everything that isn't pointed upward. Useful in its own way, but not what you need. I'm not sure what the actual solution in VEX would be--it's not an easy problem, from what I can tell. There is, however, a Measure Curvature tool in Labs that might give you what you want.

BryanRay
I'm not sure what the actual solution in VEX would be--it's not an easy problem, from what I can tell

Observed the same as in the previous comment.

I've just started to learn a little VEX so this is just a couple of ideas.

In the list of geometry VEX functions there are neighbor and neighbors. If remembering correctly they will give you the point ids of points next to the current point.

From there you would probably need to compare the normals between your point and each point in the list.

In shading context there is occlusion, but probably this is not what you want, no?

Either curvature from Measure SOP or you can implement occlusion in VEX by firing rays and using hits.

animatrix_
Either curvature from Measure SOP or you can implement occlusion in VEX by firing rays and using hits.

I understand the firing rays, although how do I do occlusion in VEX ?

SWest

BryanRay
I'm not sure what the actual solution in VEX would be--it's not an easy problem, from what I can tell

Observed the same as in the previous comment.

I've just started to learn a little VEX so this is just a couple of ideas.

In the list of geometry VEX functions there are neighbor and neighbors. If remembering correctly they will give you the point ids of points next to the current point.

From there you would probably need to compare the normals between your point and each point in the list.

In shading context there is occlusion, but probably this is not what you want, no?

Occlusion in shading, correct; is not what I want to do

_Christopher_
although how do I do occlusion in VEX ?

you can just use Mask By Feature SOP, which has Ambient Occlusion option
if you are interested in VEX you can have a look inside, it's all in make_light_and_shadow wrangle

even though for your purposes it definitely sounds like Measure SOP curvature or one of the Labs curvature SOPs should be the first to try, there is also plenty of interesting VEX inside if you are curious

Out of curiosity I went a bit further with the original idea of comparing normals.

However, there is a problem with this, and that is how to mathematically know the difference between if a curvature is convex or concave.

Here's an image to show an example using a simple "a dot b" thing with the neighbor normals. Both the concave and the convex areas are darkened.

Edit: After investigating previous suggestions it seems like using dot product only will not work for distinguishing convex and concave curvature. Min and max curvature need to be calculated from coefficients coming from a matrix (from the point neighbors). This can be used to distinguish between convex and concave curvature. The mathematical steps required for this is advanced. A working solution is (confirmed to be) available according to the early answer from BryanRay.