Indeed in the game of chicken appearing "insane" is actually a good strategy
One tactic in the game is for one party to signal their intentions convincingly before the game begins. For example, if one party were to ostentatiously disable their steering wheel just before the match, the other party would be compelled to swerve. This shows that, in some circumstances, reducing one's own options can be a good strategy.
Pre-commitment is part of a larger class of "wastefully costly" signals.
However the "madman strategy" (linked from another answer) is not a simple game with predetermined payoffs like the chicken game. Rather, the example given there uses prisoner's dilemma as the initial game. But what it is actually saying with the -100 part is that since, in reality, the (negative) payoff for war is not known, a "madman" bluff transforms the prisoner's dilemma game into a game of chicken! (Compare the tables.)
The essential difference between these two games is that in the prisoner's dilemma, the Cooperate strategy is dominated, whereas in Chicken the equivalent move is not dominated since the outcome payoffs when the opponent plays the more escalated move (Straight in place of Defect) are reversed.
So basically the "madman strategy" is a "meta-game" trying to induce the opponent to think it's playing a different game. N.B.: after asking on econ.SE, the general term is Bayesian game. And more precisely, a signalling game. The best illustration related to your question is the so-called "reputation game" (a sub-type of signalling game):
The sender can be one of two types: Sane or Crazy. A sane sender can send one of two messages: Prey and Accommodate. A crazy sender can only Prey.
The receiver can do one of two actions: Stay or Exit. Apriori, the sender has probability p to be Sane and 1-p to be Crazy. [The payoff table is below]
The a priori numerical constraints are assumed to be
- For the Sane sender, the ordering (his utility function) is M1>D1>P1, so the successful "faking mad" monopoly choice has higher payoff (if successful, i.e. if receiver exits) than a cooperative duopoly (D1) which in turn is higher still than the payoff for just "faking mad". (The last is usually a negative number in reality, i.e. a cost, but this assumption is not strictly necessary, it just needs to be worse/less than the other two aforementioned payoffs.)
- For the receiver, D2>0>P2, so the message-receiving player has higher payoff if it holds his nerves (and pulls off a "compromise" duopoly) rather than caving in and exiting.
If preying is costly for a sane sender (D1+D1=P1+M1), they will accommodate and there will be a unique separating PBE [perfect Bayesian equilibrium]: the receiver will stay after Accommodate and exit after Prey.
A separating [perfect Bayesian] equilibrium is an equilibrium where senders with different types always choose different messages. This means that the sender's message always reveals the sender's type, so the receiver's beliefs become deterministic after seeing the message.
If preying is not too costly for a sane sender (D1+D1 is less than P1+M1), and it is harmful for the receiver (p D2 + (1-p) P2 ≤ 0), the sender will prey and there will be a unique pooling PBE: again the receiver will stay after Accommodate and exit after Prey. Here, the sender is willing to lose some value by preying in the first period, in order to build a reputation of a predatory firm, and convince the receiver to exit.
A pooling [perfect Bayesian] equilibrium is an equilibrium where senders with different types all choose the same message. This means that the sender's message does not give any information to the receiver, so the receiver's beliefs are not updated after seeing the message.
The latter is the "madman strategy". It takes specific payoff constraints and player-type initial distribution to be the optimal strategy even for the Sane type. And for completeness, there's one more case:
If preying is not costly for the sender nor harmful for the receiver, there will not be a PBE in pure strategies. There will be a unique PBE in mixed strategies - both the sender and the receiver will randomize between their two actions.
The latter case, i.e. occasional (non-permanent) use a of "madman strategy" is actually more related to strategic ambiguity, as are all mixed strategies. I'm handwaving the details on the latter here; strategic ambiguity actually has a more precise meaning, withholding information in order to create game-shifting a situation similar to "playing madman", but without explicitly sending unambiguous "crazy" messages:
strategic ambiguity can be used
by a mediator to achieve cooperation in situations similar to the prisoners’
dilemma. In this game a mediator is able to create ambiguity about the
reward in case of unilateral defection. If he creates enough ambiguity, both
prisoners are afraid of punishment and prefer to cooperate. The outcome
thus reached is even Pareto–improving. A remarkable consequence [...] is that the strategic use
of ambiguity allows to reach equilibria that are not Nash equilibria in the
original game [...].
Under the assumption that receivers are ambiguity-averse, sending ambiguous messages can thus obtain a similar effect to "playing madman"... and is much more "appearances saving". Although strategic ambiguity has been formalized much more recently in game theory than the explicit "madman strategy", diplomats have been using the former for probably as long as they have been using the more naked "playing madman" strategy.