Hauser considers the “scariest part” of Rip Wheeler’s demeanor to be how “normal” he is

Cole Hauser’s true grit cowboy persona on Yellowstone has become one of the most defining elements of the hit Paramount Network series throughout its five-season run. However, the actor initially believed that his character was “going to be so hated” by viewers.

Hauser, 49, takes on the role of jaded ranch foreman Rip Wheeler in the western drama. Rip first came to John Dutton (Kevin Costner) as a young boy who was on the run after killing his stepfather to protect his mother. John took him in like a son and taught him how to be a cowboy, while also frequently leveraging his hot streak to exact merciless violence and murder against anyone who went against the ranch.

Hauser tells PEOPLE that fans’ love for Rip “surprised the s— out of [creator] Taylor Sheridan and I, at first,” particularly noting what makes him such a menacing character.

The star compares Rip to being “like normal people. And I think that’s what’s the scariest part,” adding that “he is just that kind of animal.”

Furthermore, Hauser notes that there are parallels between the “bad guys” he plays, particularly how they “don’t really play themselves as bad guys.”

That awareness of Rip’s menacing nature led Hauser to question why fans, particularly women, were so enamored by his character.

“At first, to see women coming up and saying, ‘Oh my God, I love your character.’ I was thinking, ‘What’s wrong with you?’” he quips but explains that his mentality on the subject changed once he understood fans’ perspective a bit more.

Hauser says that as Yellowstone has “grown [in popularity], I see the bad boy kind of thing that they love.”

“What makes [Rip] special is that he has an unbelievable heart too,” the actor adds. “I mean, his loyalty is to not only the Dutton family, but to Beth [Dutton (Kelly Reilly)] and his kindness, other than wanting to kill people times, is just, to me, it’s the best of both worlds to be able to play.”

Hauser adds that the emotional depth of Rip as a character is, in part, what has kept him interested in the role throughout the duration of Yellowstone.

“You look for colors in characters, and Taylor, in the last five seasons, has given Rip those wonderful colors,” he concludes.