Researchers at Sogang University have developed a technique that can cut the response time of large language models roughly in half, a breakthrough that could improve the efficiency of AI systems that power chatbots and other generative tools.
The study, led by professor Yi Young-min and graduate student Song Geun-soo from the university’s Department of Artificial Intelligence, has been accepted to the international computer architecture conference ASPLOS 2026, the university said Tuesday.
Their paper proposes a method called oFFN, short for Outlier and Neuron-aware Structured Feedforward Network, designed to accelerate the most computation-intensive layer inside large language models.
In modern AI models, a large portion of the processing workload occurs in the feedforward network layer, where massive calculations are performed to generate each token in a response. The researchers found that certain extreme values, known as “outliers,” tend to appear in specific positions within these calculations.
Traditional AI systems perform every calculation because it is difficult to predict which operations are essential. The new method reorganizes these computations by identifying where outliers and frequently used neurons appear and arranging them in an order that allows the system to process them more efficiently.
Tests showed that the technique increased the processing speed of the feedforward network by as much as 5.46 times. Users' perceived the total response generation time to be up to twice as good.
Despite the faster performance, the researchers said the system maintained nearly the same level of accuracy and response quality. The method also ran about 13 percent faster than existing state-of-the-art approaches.
Yi said the study demonstrates how GPU tensor cores and CUDA cores can be used in complementary ways to accelerate inference in both single-batch and multi-batch scenarios.
ASPLOS, short for the International Conference on Architectural Support for Programming Languages and Operating Systems, is considered one of the leading conferences in computer architecture and systems. This year's event will be held in Pittsburgh from Sunday until March 26.
hnpark@heraldcorp.com
