formula – Regex Validation Rule Help

Normal regex uses a single backslash to escape a character, but in Salesforce, we need to double each backslash.

You also need to escape the hyphen (-) after the apostrophe (because it has a special meaning inside of square braces).

So I believe you’re looking for NOT(REGEX(FirstName, "[a-zA-Z'\-\.]+")). I don’t think that making the overall regex a capturing group does anything for you here (which is what the ( and ) enclosing the entire regex does). I don’t think the ending anchor $ does anything for you either.

As for the “only one apostrophe” part, I’d think that’d be easiest to achieve with a separate regex, so your validation rule would end up looking like

AND(
    NOT(REGEX(<first one>)),
    NOT(REGEX(<second one>))
)

Regex is only capable of some pretty basic/simple counting, but something like '.*' ought to do (apostrophe followed by 0 or more other characters, followed by an apostrophe).