WASHINGTON -- Rep. Louie Gohmert (R-Texas), one of five House Republicans who claimed the Muslim Brotherhood may have infiltrated the government, promised Thursday that he doesn't accuse anyone of nefarious behavior just because they are Muslim.

"You follow me around the world, you see me hugging Muslims around the world, because the ones I hug are our friends," Gohmert told Department of Homeland Security Secretary Janet Napolitano during a House Judiciary Committee meeting. "This administration seems to have a hard time recognizing members of terrorist groups who are allowed into the White House."

Gohmert is in the middle of a firestorm this week after he joined Rep. Michele Bachmann (R-Minn.) and others in claiming that former State Department aide Huma Abedin, the wife of former Rep. Anthony Weiner (D-N.Y.), may have ties to the Muslim Brotherhood.

Those accusations were roundly condemned by other GOP members, including by House Speaker John Boehner (R-Ohio) -- he called the claims "pretty dangerous" -- and Sens. John McCain (R-Ariz.) and Scott Brown (R-Mass.).

On Thursday, Gohmert questioned Napolitano on Mohamed Elibiary, an appointee on President Obamas Homeland Security Advisory Council who was accused in 2011 of leaking intelligence documents to make Texas Gov. Rick Perry look Islamaphobic.

Napolitano told him she'd looked into the matter -- and that Gohmert's claims about Elibiary were inaccurate and that he did not, to her knowledge, download classified documents. She also took a shot at Gohmert's talk about Muslims, though without specifically referencing the recent claims about the Muslim Brotherhood.

"What bothers me, quite frankly, are the allegations that remain against anyone who happens to be Muslim," Napolitano said.

"The allegations are not because he is Muslim," Gohmert replied.

As the discussion continued, the congressman raised his voice, and Judiciary Committee Chairman Lamar Smith (R-Texas) repeatedly attempted to make him tone it down -- or stop altogether -- as Napolitano remained calm.

"I'm asking you if you know simple facts," he shouted at one point, interrupting her. He later accused her of avoiding answers several times as she was trying to speak.

"This committee has a long and proud tradition, and these kinds of insinuations demean the committee," Napolitano said. "The insinuation that I or my staff would allow someone who is a terrorist to infiltrate --"

He broke in to say he was making "no such insinuation" and that Elibiary is "a very nice gentleman."

At that point, Smith told him his time was up.