Income buffer is to small for a payload sent with QoS1

You don’t say which version of MQTT you are using.

If you’re using MQTT 3.1.1, you’re kind-of stuck. MQTT 3.1.1 provides no “maximum packet size”, and it provides no way to reject a QoS 1/2 PUBLISH packet. You’re forced to disconnect and reconnect with “Clean Session = 1”, to purge the too-big message (and everything else) from the broker’s session state for your client.

If you’re using MQTT 5, you can specify the “Maximum Packet Size”. (Not the “Receive Maximum”: the Receive Maximum controls the *number* of QoS 1/2 PUBLISH packets the broker is allowed to send to you before acknowledgment, not the packet *size*.) If the broker then insists on sending you a packet larger than the Maximum Packet Size, you’re supposed to disconnect the connection with Reason Code 0x95 (packet too large). But, at that point, you’re dealing with a non-conformant broker, so all bets on broker behaviour are off.

In MQTT 5, you can also reject individual PUBLISH by sending PUBACK (QoS 1) or PUBREC (QoS 2) for the Packet Identifier with a Reason Code ≥ 0x80, in which case the broker is not supposed to ever retransmit the PUBLISH of those messages. But, 0x95 is not an allowed Reason Code for PUBACK or PUBREC.