|"You did not come home when you said that you would," has as yet to say the actual fact.
Saying to someone "what did NOT occur," rather than the actual historical event that DID occur, can suggest blame or criticism to the hearer, possibly to be received and defended against as some sort of a "should", or a "judgment."
Similar VRP statements are: You didn't sign out. You never said that before. I didn't get your call. You didn't send flowers.
||"You came home two hours after you said that you would be here," is a statement containing two Observations: The first: When he or she did come home; and The second: The relationship between the coming-home time and the time that the person said that he or she would arrive.