The listeners could be the ghosts of the people who used to live in the house. The Traveller is able to feel their presence, and in a strange, almost disconcerting way, hear their silence, but sees no evidence of movement in the house; however, he also does not feel like the house is empty either. Even people who at being incredibly quiet make a discernible noise but here the overwhelming sound is silence. The ghosts of those once living in the house still make it feel as though something inhabits it. The poet also calls the Traveller "the voice from the world of men" which suggests that the listeners are no longer part of that world.
People often think of language as a connector, something that brings people together by helping them share experiences, feelings, ideas, etc. We, however, are interested in how language sets people apart. Start with the peculiarities of your own personal language—the voice you use when speaking most intimately to yourself, the vocabulary that spills out when you're startled, or special phrases and gestures that no one else seems to use or even understand—and tell us how your language makes you unique. You may want to think about subtle riffs or idiosyncrasies based on cadence, rhythm, rhyme, or (mis)pronunciation.
–Inspired by Kimberly Traube