IF this property is not present on your instance, then you need to create this system property and give limit as per your requirement.

But, this will be applicable for all multi row variable sets.

IF you want to limit number of rows only for particular catalog item, then you need to write onsubmit client script to count number of rows with validation.

Beliw example  is a list of serial numbers multi row variable set, where serial_number is the internal name of the MRVS

var snlist = g_form.getValue('serial_number');
var obj = JSON.parse(snlist);
var length = obj.length;

Regards,

Sachin