What does Mr.Lamb mean by saying that "No, Not the whole of you. Not of you." ?

Mr. Lamb says that he is not afraid of the whole of Derry. His disfigured face creates dread but it is that part of Derry which refuses to look past his disfigurement that Lamb points out as being actually scary. Unless Derry stops his self-loathing, there is not hope for him to actually understand that there are parts of him which are beautiful in spite of his physical disfigurement.