"Childhood is immense in the development of the later writer," he adds. "The deck was the formative principle of my becoming a writer."
Born to a Portuguese father from Shanghai and a Chinese-English mother, Castro started writing diaries at 15.
His mother was a traditional Chinese housewife who came from a village in southern Guangdong and who read a lot.
"My grandfather cast influence on me through my mother to be a scholar in the ancient Chinese way," he says. "So reading was important in the children of my family."
Castro's businessman father was also a musician whom the author credits with infusing musical rhythm in his writing.
In 1982, Castro published his debut novel Birds of Passage. Since then, China, its places and its people have been offering constant background and inspiration to the writer.
"I always start with specific places. In writing, I do a lot for atmosphere, like the smell, sound and environment. I want readers to feel the places," he says.
Often paying homage to modern writers from other cultures in his works, Castro says James Joyce is his recurring reference point, like in the iconic one-page long sentences.
Castro enjoys putting old stories in a modern context, as Joyce did in Ulysses.
"Shanghai, Hong Kong and Macao were the cities I like to remember in the way Joyce remembers his hometown Dublin," he says.
But Joyce never got back to home. Castro did. Castro was first brought back to Shanghai in 1992, the year of his father's death.
In 1932, his father moved from Shanghai to Hong Kong due to the tension of forthcoming war.
"I was just there to trace his memory," he says. "I went to stay in the Peace Hotel, the jazz band reminded me of him."
And later, he developed Shanghai Dance from the memories.
Castro's later visits were mostly for literary exchanges with Chinese writers.
A literary critic who's also fluent in French, Cantonese and Portuguese, Castro finds the gap between literature and its translation interesting.
He has visited Shanghai for four times and Beijing twice, through the "lively and dynamic exchanges", he sees "how much we have things in common, but how much we're different".
He enjoys Mo Yan's Sandalwood Death and finds a Shakespearean presence in Mo Yan's works, together with "interesting romantic moments with some kind of irony".
Works by classical writer Lu Xun and contemporary writer Han Shao-gong also appeal to him. "Lu Xun is an interesting man in his personal life. And his works show how he walked out of a depressing and hard life to make such a difference," he says.
Through the literary exchanges with Chinese writers, Castro is surprised to find that "Chinese actually put more emphasis on literature than the Australians" because of the huge reading population and the sound governmental support.
"Australia has smaller reading population on literature, and the literary circle is small," he says.
Though readers like Australian Sinologist Colin Macherras say they enjoy Castro's writing, his works sometimes are still considered "difficult reading" in the country.
"They take difficulty as dismissal," Castro says. "They also use 'difficulty' as complexity."
Castro says in the country with many immigrants, people tend to classify things. People want something that is black-and-white, and want literature to be easily understood.
"When it can't be classified, they call it difficult," he says.
"I'm a complex person, coming from different cultures," he says. "Complexity is a good thing."