Modern Deep Learning technologies such as Self Attention and RNN, kind of integrated higher-degree nonlinear functions fundamentally. e.g. $(w_{1_{branch1}}x_1 + w_{2_{branch1}}x_2) \cdot (w_{1_{branch2}}x_1 + w_{2_{branch2}}x_2)$ .

Besides, there aren't too much development on e.g. trigonometric function AFAIK. From Bionics perspective, is there any nonlinear-functional neuron actually exists in human brain? If so, what kind of exact function do they encode?

Related question:

Is there a biological equivalent to the bias term used in artificial neural networks?

What are the rules that govern neuron behavior?

  • $\begingroup$ All biological neurons are nonlinear. $\endgroup$
    – Bryan Krause
    Feb 11 at 15:28
  • $\begingroup$ @BryanKrause Are they all unable to be ideally treated as linear? If so, what is the similarity between Artificial Neural Network and "real neural network"? Apologize for my limited understanding about biology. $\endgroup$ Feb 11 at 16:23
  • 1
    $\begingroup$ Not very similar, just loosely motivated. If you want to learn about neurobiology, I'd recommend starting from the basics and learning bottom-up, rather than starting from what you know about ANN/ML and working backwards making assumptions. In the "all models are wrong but some are useful" sense we might sometimes treat biological neurons as linear in order to more easily simulate populations to test particular assumptions, but it's sort of a "take a uniform spherical cow" situation in physics where everyone knows the model is wrong. $\endgroup$
    – Bryan Krause
    Feb 11 at 16:46
  • $\begingroup$ @BryanKrause Didn't know "spherical cow", thanks for your informative replay though. These well working models maybe far from "correct". $\endgroup$ Feb 11 at 17:45
  • 1
    $\begingroup$ There are at least two categories of neuron "model": those that are useful for learning about the brain and those that are useful for solving other tasks. They're pretty much mutually exclusive, because the things that are mechanistically easy with biology are computationally really expensive, and vice versa. Artificial networks tend to become less and less brain-like the more they are optimized to solve tasks (I.e. machine learning/AI) $\endgroup$
    – Bryan Krause
    Feb 11 at 18:26


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.