How do psychologists measure malicious behaviour in humans? Is there a definitive definition of what constitutes malicious behaviour? Are there varying scales of malicious behaviour? Is it possible do conduct experiments where malicious behaviour can be reliably observed? E.g. is cheating in a competition against other humans considered malicious? If malicious behaviour is much further along the spectrum than this example, then how can you conduct such an experiment that remains ethical?
"Malicious behavior" is not a term with a specific definition in psychology; if you search Google Scholar for "malicious behavior" + psychology you receive relatively few results and I see no evidence that those papers are using it as a term, rather they use it as a description, that is, the words are used simply in the context of their meaning in English.
Any single study that wishes to investigate malicious behavior would need to define it operationally for that specific study, and it would no doubt depend on the aims and subjects of the study. Malicious behavior in toddlers would almost certainly be defined differently than in adults; malicious behavior in the workplace would be defined differently than malicious behavior in the context of some psychiatric disorder.