Wins: Discovering Efficient Activation Functions for Sparse LLMs
The paper indicates through experiments that models employing excel across all three evaluation aspects, highlighting its potential as an efficient activation function for sparse LLMs.