How do psychologists measure malicious behaviour in humans? Is there a definitive definition of what constitutes malicious behaviour? Are there varying scales of malicious behaviour? Is it possible do conduct experiments where malicious behaviour can be reliably observed? E.g. is cheating in a competition against other humans considered malicious? If malicious behaviour is much further along the spectrum than this example, then how can you conduct such an experiment that remains ethical?
The Dark Triad maybe what you are seeking.
You have described the behaviours as "malicious", where as the Dark Triad uses the word "malevolent". The Oxford Dictionary advises that these two words are synonyms.
Jones, D. N., & Paulhus, D. L. (2014). Introducing the Short Dark Triad (SD3): A Brief Measure of Dark Personality Traits. Assessment, 21(1), 28–41. https://doi.org/10.1177/1073191113514105
"Malicious behavior" is not a term with a specific definition in psychology; if you search Google Scholar for "malicious behavior" + psychology you receive relatively few results and I see no evidence that those papers are using it as a term, rather they use it as a description, that is, the words are used simply in the context of their meaning in English.
Any single study that wishes to investigate malicious behavior would need to define it operationally for that specific study, and it would no doubt depend on the aims and subjects of the study. Malicious behavior in toddlers would almost certainly be defined differently than in adults; malicious behavior in the workplace would be defined differently than malicious behavior in the context of some psychiatric disorder.
Here is a scale on benign and malicious envy. It isn't exactly what you want, but it may contain some citations that are of interest you. I believe they also mention the dark triad, which @Tony Mobbs suggested. Both are worth a look at least. I think the scale is on OSF (osf.io) if you can't access the paper.
Lange, J., & Crusius, J. Dispositional envy revisited: Unraveling the motivational dynamics of benign and malicious envy. Personality and Social Psychology Bulletin, 41(2), 284-294. https://doi.org/10.1177/0146167214564959