Saw this video before asking, but didn't read the comments, which mentioned a solution to the problem without UVs.
Just in case anybody wonders how to do it without uv's, in case you are doing some proceduraly textureing, add a rest position node before any animation takes place, kind of the same but you won't have to create uv's - Max Rodas
This is caused because the Noise doesn't know how to compute the point deformation from the rubber toy, this kind of case is solved by just adding a rest attribute (only for constant topology).