Sean Penn still remembers when he heard the rumor that accused him of hitting Madonna with a baseball bat.

Allegations of domestic violence, which Madonna has since denied, were in the news during and after the couples rocky marriage from 1985 to 1989. However, Penn said he first heard the particular story about the bat over a morning cigarette.

A woman he was dating confronted him about it after a lovely evening together, he told The New York Times in an interview that published Saturday.

Shes looking at me like I killed her dog, the Mystic River star said.

I didnt know what the hell she was talking about, the Oscar winner added. Now I think its fair to say that Im not the biggest guy in the world. But if I hit Mike Tyson in the head with a baseball bat, hes going to the hospital.

The rumor that Penn hit the singer with a bat in 1987 fit in with the narrative around the pairs relationship that flared up when Madonna filed an assault complaint against Penn in December 1988. While she reportedly withdrew the complaint a week later, police flooded the couples Malibu, California, home and ordered Penn outside with his hands up.

I had a freaking SWAT team come into my house, the 63-year-old actor told the Times. I said: Im not coming out. Im going to finish my breakfast. The next thing I knew, windows were being broken all around the house and they came in.

The allegations of abuse drew attention again in 2015 when director Lee Daniels defended his Empire star Terrence Howard by comparing the actors own admitted domestic violence to Penns alleged actions with Madonna .