I was able to consult some other Labkey devs and got this working. If anyone else is wondering about this - the "error" parameter in the trigger functions is to be set and will be displayed to the user. For example:

function beforeInsert(row, errors){
    errors[null] = 'Don't feel like inserting this record rn'
}

will result in a record not getting inserted and the user seeing the message in the UI.