how did the author come to know that she has come to the right address?

On her first visit to the address, she was quite unsure as to whether she had reached the correct address or not. However, when she saw Mrs. Dorling wearing her mother's green knit cardigan with the colour of the buttons turning pale, she instantly recognised it and thus, got convinced that she had reached the right address.