EfficientPaper

Pruning, Quantization and efficient-inference/training paper list.

Table of Contents

• EfficientPaper
• Getting Started
• Paper List
  • keyword
  • year
  • publication
  • institution
  • author
• References

Getting Started

git clone https://github.com/hustzxd/EfficientPaper
pip install protobuf==5.27.2 pandas arxiv 

1. Add paper information by ./add_paper_info.sh
2. Run ./refresh_readme.sh

paper {
  title: "EfficientPaper: manage your research papers in an efficient way."
  abbr: "EfficientPaper"
  url: "https://github.com/hustzxd/EfficientPaper"
  authors: "hustzxd"
}
pub {
  where: "GitHub"
  year: 2023
}
code {
  type: "Pytorch"
  url: "https://github.com/hustzxd/EfficientPaper"
}
note {
  url: "EfficientPaper.md"
}
keyword {
  words: efficient_paper
}

Paper List

year

2025

1. AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inference []
2. Pruning Large Language Models with Semi-Structural Adaptive Sparse Training []
3. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention []
4. COMET: Towards Partical W4A4KV4 LLMs Serving []
5. POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference []
6. vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention []
7. BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity []
8. KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs []
9. Enhancing One-shot Pruned Pre-trained Language Models through Sparse-Dense-Sparse Mechanism []
10. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion []
11. FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference []
12. Forgetting Transformer: Softmax Attention with a Forget Gate []
13. R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference []
14. ReAttention: Training-Free Infinite Context with Finite Attention Scope []
15. Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA []
16. Training-Free Activation Sparsity in Large Language Models []
17. AdaSplash: Adaptive Sparse Flash Attention []
18. BaWA: Automatic Optimizing Pruning Metric for Large Language Models with Balanced Weight and Activation []
19. CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration []
20. Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity []
21. HashAttention: Semantic Sparsity for Faster Inference []
22. La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation []
23. MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attention []
24. ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference []
25. SlimLLM: Accurate Structured Pruning for Large Language Models []
26. SpargeAttn: Accurate Sparse Attention Accelerating Any Model Inference []
27. Sparsing Law: Towards Large Language Models with Greater Activation Sparsity []
28. Star Attention: Efficient LLM Inference over Long Sequences []
29. XAttention: Block Sparse Attention with Antidiagonal Scoring []
30. TorchAO: PyTorch-Native Training-to-Serving Model Optimization []
31. AMALI: An Analytical Model for Accurately Modeling LLM Inference on Modern GPUs []
32. SpecEE: Accelerating Large Language Model Inference with Speculative Early Exiting []
33. Unifying Mixture of Experts and Multi-Head Latent Attention for Efficient Language Models []
34. NanoFlow: Towards Optimal Large Language Model Serving Throughput []
35. A Simple Linear Patch Revives Layer-Pruned Large Language Models []
36. Acc-SpMM: Accelerating General-purpose Sparse Matrix-Matrix Multiplication with GPU Tensor Cores []
37. Accelerating LLM Inference Throughput via Asynchronous KV Cache Prefetching []
38. Accelerating Prefilling for Long-Context LLMs via Sparse Pattern Sharing []
39. Adaptive Computation Pruning for the Forgetting Transformer []
40. Adaptive Layer-skipping in Pre-trained LLMs []
41. AhaKV: Adaptive Holistic Attention-Driven KV Cache Eviction for Efficient Inference of Large Language Models []
42. Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Models []
43. AttentionPredictor: Temporal Pattern Matters for Efficient LLM Inference []
44. CCQ: Convolutional Code for Extreme Low-bit Quantization in LLMs []
45. Characterizing Communication Patterns in Distributed Large Language Model Inference []
46. Characterizing Compute-Communication Overlap in GPU-Accelerated Distributed Deep Learning: Performance and Power Implications []
47. ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference []
48. Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts []
49. DBudgetKV: Dynamic Budget in KV Cache Compression for Ensuring Optimal Performance []
50. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning []
51. Delta Attention: Fast and Accurate Sparse Attention Inference by Delta Correction []
52. DeltaLLM: A Training-Free Framework Exploiting Temporal Sparsity for Efficient Edge LLM Inference []
53. Efficient LLM Inference: Bandwidth, Compute, Synchronization, and Capacity are all you need []
54. Efficient Long-Decoding Inference with Reasoning-Aware Attention Sparsity []
55. Exploiting Sparsity for Long Context Inference: Million Token Contexts on Commodity GPUs []
56. Fast and Simplex: 2-Simplicial Attention in Triton []
57. FastKV: KV Cache Compression for Fast Long-Context Processing with Token-Selective Propagation []
58. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving []
59. FlashOverlap: A Lightweight Design for Efficiently Overlapping Communication and Computation []
60. FreqKV: Frequency Domain Key-Value Compression for Efficient Context Window Extension []
61. HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention for Scalable Large Model Inference []
62. HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs []
63. Hardware-Efficient Attention for Fast Decoding []
64. Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding []
65. Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregation []
66. Instruction-Following Pruning for Large Language Models []
67. KV Cache Compression for Inference Efficiency in LLMs: A Review []
68. KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse []
69. KeepKV: Eliminating Output Perturbation in KV Cache Compression for Efficient LLMs Inference []
70. LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention []
71. LeanK: Learnable K Cache Channel Pruning for Efficient Decoding []
72. MIRAGE: KV Cache Optimization through Parameter Remapping for Multi-tenant LLM Serving []
73. MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production []
74. MiniCPM4: Ultra-Efficient LLMs on End Devices []
75. Mixture of Experts in Large Language Models []
76. Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing []
77. Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation []
78. MoBA: Mixture of Block Attention for Long-Context LLMs []
79. Mosaic: Composite Projection Pruning for Resource-efficient LLMs []
80. Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs []
81. PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attention []
82. Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving []
83. Pruning General Large Language Models into Customized Expert Models []
84. QuickSilver -- Speeding up LLM Inference through Dynamic Token Halting, KV Skipping, Contextual Token Fusion, and Adaptive Matryoshka Quantization []
85. Qwen3 Technical Report []
86. R-KV: Redundancy-aware KV Cache Compression for Training-Free Reasoning Models Acceleration []
87. Radial Attention: $O(n\log n)$ Sparse Attention with Energy Decay for Long Video Generation []
88. Rectified Sparse Attention []
89. Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving []
90. SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling []
91. SEAP: Training-free Sparse Expert Activation Pruning Unlock the Brainpower of Large Language Models []
92. SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training []
93. SeerAttention-R: Sparse Attention Adaptation for Long Reasoning []
94. Seesaw: High-throughput LLM Inference via Model Re-sharding []
95. Speed Always Wins: A Survey on Efficient Architectures for Large Language Models []
96. SpindleKV: A Novel KV Cache Reduction Method Balancing Both Shallow and Deep Layers []
97. Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding []
98. Task-KV: Task-aware KV Cache Optimization via Semantic Differentiation of Attention Heads []
99. The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs []
100. TileLink: Generating Efficient Compute-Communication Overlapping Kernels using Tile-Centric Primitives []
101. TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inference []
102. Triton-distributed: Programming Overlapping Kernels on Distributed AI Systems with the Triton Compiler []
103. Unveiling Super Experts in Mixture-of-Experts Large Language Models []
104. Attention-Gym: Triton-Based Sparse and Quantization Attention []
105. Unified KV Cache Compression Methods for Auto-Regressive Models []
106. kvpress: LLM KV cache compression made easy []

2024

1. Fluctuation-based Adaptive Structured Pruning for Large Language Models []
2. ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition []
3. Centauri: Enabling Efficient Scheduling for Communication-Computation Overlap in Large Model Training via Communication Partitioning []
4. T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectives []
5. Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention []
6. A novel CUTLASS-based implementation of Tensor Parallelism for NVLink-enabled systems []
7. CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models []
8. Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption []
9. SparseInfer: Training-free Prediction of Activation Sparsity for Fast LLM Inference []
10. Post-Training Statistical Calibration for Higher Activation Sparsity []
11. A Simple and Effective Pruning Approach for Large Language Models []
12. Compressing LLMs: The Truth is Rarely Pure and Never Simple []
13. Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs []
14. Efficient Streaming Language Models with Attention Sinks []
15. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning []
16. Plug-and-Play: An Efficient Post-training Pruning Method for Large Language Models []
17. QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models []
18. SAS: Structured Activation Spasification []
19. SliceGPT: Compress Large Language Models by Deleting Rows and Columns []
20. ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models []
21. Accelerating Transformer Pre-training with 2:4 Sparsity []
22. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty []
23. FrameQuant: Flexible Low-Bit Quantization for Transformers []
24. LoRA+: Efficient Low Rank Adaptation of Large Models []
25. OSSCAR: One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimization []
26. Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity []
27. Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference []
28. SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models []
29. SparQ Attention: Bandwidth-Efficient LLM Inference []
30. Sparse is Enough in Fine-tuning Pre-trained Large Language Models []
31. Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency []
32. SqueezeLLM: Dense-and-Sparse Quantization []
33. TinyTrain: Resource-Aware Task-Adaptive Sparse Training of DNNs at the Data-Scarce Edge []
34. Unleashing the Power of Meta-tuning for Few-shot Generalization Through Sparse Interpolated Experts []
35. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration []
36. Vidur: A Large-Scale Simulation Framework For LLM Inference []
37. MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention []
38. MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models []
39. SGLang: Efficient Execution of Structured Language Model Programs []
40. SlimGPT: Layer-wise Structured Pruning for Large Language Models []
41. SparseLLM: Towards Global Pruning for Pre-trained Language Models []
42. Fast and Effective Weight Update for Pruned Large Language Models []
43. Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity []
44. A Survey on Efficient Inference for Large Language Models []
45. A Survey on Inference Optimization Techniques for Mixture of Experts Models []
46. A Survey on Large Language Model Acceleration based on KV Cache Management []
47. APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Serving []
48. AVSS: Layer Importance Evaluation in Large Language Models via Activation Variance-Sparsity Analysis []
49. Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference []
50. Beyond 2:4: exploring V:N:M sparsity for efficient transformer inference on GPUs []
51. Beyond KV Caching: Shared Attention for Efficient LLMs []
52. Compact Language Models via Pruning and Knowledge Distillation []
53. CoreInfer: Accelerating Large Language Model Inference with Semantics-Inspired Adaptive Sparse Activation []
54. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model []
55. DeepSeek-V3 Technical Report []
56. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models []
57. Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping []
58. DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads []
59. Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment []
60. Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes []
61. FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion []
62. FlashMask: Efficient and Rich Mask Extension of FlashAttention []
63. Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache []
64. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization []
65. L4Q: Parameter Efficient Quantization-Aware Training on Large Language Models via LoRA-wise LSQ []
66. LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning []
67. LLM Inference Serving: Survey of Recent Advances and Opportunities []
68. Massive Activations in Large Language Models []
69. MiniKV: Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache []
70. Mixture-of-Depths: Dynamically allocating compute in transformer-based language models []
71. MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression []
72. Multi-matrix Factorization Attention []
73. Optimizing LLM Inference via Channel-Wise Thresholding and Selective Sparsification [] Pytorch
74. Post-Training Sparse Attention with Double Sparsity []
75. PowerInfer-2: Fast Large Language Model Inference on a Smartphone [] Website
76. ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models []
77. Q-Sparse: All Large Language Models can be Fully Sparsely-Activated []
78. QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving [] Pytorch
79. ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMs []
80. ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing []
81. Recycled Attention: Efficient inference for long-context language models []
82. Reducing Transformer Key-Value Cache Size with Cross-Layer Attention []
83. Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark []
84. SCBench: A KV Cache-Centric Analysis of Long-Context Methods []
85. SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization []
86. SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration []
87. SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attention []
88. SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs []
89. ShadowLLM: Predictor-based Contextual Sparsity for Large Language Models []
90. SnapKV: LLM Knows What You are Looking for Before Generation []
91. Transformers are Multi-State RNNs []
92. Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters [] Pytorch
93. XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models []
94. ZigZagkv: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertainty []
95. [Distributed w/ TorchTitan] Introducing Async Tensor Parallelism in PyTorch []

2023

1. Diffuser: Efficient Transformers with Multi-hop Attention Diffusion for Long Sequences []
2. Gradient-based Intra-attention Pruning on Pre-trained Language Models []
3. Pruning Pre-trained Language Models Without Fine-Tuning []
4. Pruning Pre-trained Language Models with Principled Importance and Self-regularization []
5. Structured Pruning for Efficient Generative Pre-trained Language Models []
6. Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models []
7. Structural Pruning of Large Language Models via Neural Architecture Search []
8. SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformer []
9. TorchSparse++: Efficient Point Cloud Engine []
10. AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning []
11. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers []
12. Minimum Variance Unbiased N:M Sparsity for the Neural Gradients []
13. The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers []
14. Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time []
15. SparseGPT: Massive Language Models Can be Accurately Pruned in one-shot. []
16. Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation []
17. Efficient GPU Kernels for N:M-Sparse Weights in Deep Learning []
18. ZipLM: Inference-Aware Structured Pruning of Language Models []
19. VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Cores []
20. Efficient Memory Management for Large Language Model Serving with PagedAttention []
21. Efficient Methods for Natural Language Processing: A Survey []
22. SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models []
23. A Survey on Evaluation of Large Language Models []
24. A Survey on Model Compression for Large Language Models []
25. Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models []
26. CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X []
27. Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Models []
28. Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers []
29. Efficient Guided Generation for Large Language Models []
30. Fine-Tuning Language Models with Just Forward Passes []
31. Flash-Decoding for long-context inference []
32. Gradient-Free Structured Pruning with Unlabeled Data []
33. H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models []
34. Knowledge-preserving Pruning for Pre-trained Language Models without Retraining []
35. LLM in a flash: Efficient Large Language Model Inference with Limited Memory []
36. LLM-Pruner: On the Structural Pruning of Large Language Models []
37. LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery []
38. LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models []
39. OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models []
40. Post-training Quantization for Neural Networks with Provable Guarantees []
41. PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU []
42. Pruning Large Language Models via Accuracy Predictor []
43. QLoRA: Efficient Finetuning of Quantized LLMs []
44. QuIP: Quantization with Incoherence Processing []
45. RPTQ: Reorder-based Post-training Quantization for Large Language Models []
46. Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning []
47. SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression []
48. Sparse Fine-tuning for Inference Acceleration of Large Language Models []
49. Sparse Iso-FLOP Transformations for Maximizing Training Efficiency []
50. Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging []
51. Ten Lessons We Have Learned in the New Sparseland: A Short Handbook for Sparse Neural Network Researchers []
52. The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter []
53. Training Transformers with 4-bit Integers []
54. Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filtering []
55. ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation []
56. FasterTransformer []

2022

1. Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm []
2. TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models []
3. Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads []
4. Creating Sparse GPT-3 Models with Iterative Pruning []
5. LoRA: Low-rank adaptation of large language models []
6. SPDY: Accurate Pruning with Speedup Guarantees []
7. Sparse Attention Acceleration with Synergistic In-Memory Pruning and On-Chip Recomputation []
8. A Fast Post-Training Pruning Framework for Transformers []
9. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness []
10. Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning []
11. ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers []
12. Two Sparsities Are Better Than One: Unlocking the Performance Benefits of Sparse-Sparse Networks []
13. Transformer Acceleration with Dynamic Sparse Attention []
14. An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers []
15. The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models []

2021

1. Post-training deep neural network pruning via layer-wise calibration []
2. BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction []
3. Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch []
4. A Greedy Algorithm for Quantizing Neural Networks []
5. Channel Permutations for N:M Sparsity []
6. Accelerating Sparse Deep Neural Networks []
7. Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks []

2020

1. Fast Sparse ConvNets []
2. Inducing and Exploiting Activation Sparsity for Fast Neural Network Inference []
3. Movement Pruning: Adaptive Sparsity by Fine-Tuning []
4. GPU Kernels for Block-Sparse Weights []

2019

1. ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training []

2018

1. A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers []

2017

1. DSD: Dense-Sparse-Dense Training for Deep Neural Networks []
2. Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon []

2016

1. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding []

1993

1. Optimal Brain Surgeon and general network pruning []

1989

1. Optimal Brain Damage []
2. Optimal Brain Damage []

References

Hot

1. https://github.com/horseee/Awesome-Efficient-LLM
2. https://github.com/DefTruth/Awesome-Diffusion-Inference
3. https://github.com/DefTruth/Awesome-LLM-Inference
4. https://github.com/AmberLJC/LLMSys-PaperList
5. https://github.com/Hannibal046/Awesome-LLM
6. https://github.com/AmadeusChan/Awesome-LLM-System-Papers
7. https://github.com/KnowingNothing/compiler-and-arch
8. https://papercopilot.com/paper-list
9. https://github.com/TreeAI-Lab/Awesome-KV-Cache-Management
10. https://github.com/October2001/Awesome-KV-Cache-Compression

Cold

1. https://github.com/he-y/Awesome-Pruning
2. https://github.com/htqin/awesome-model-quantization
3. https://github.com/csyhhu/Awesome-Deep-Neural-Network-Compression
4. https://github.com/AojunZhou/Efficient-Deep-Learning
5. https://github.com/chester256/Model-Compression-Papers