Probe-Then-Commit: Constrained Reinforcement Learning for Adaptive Precision and Early-Exit Inference on Resource-Constrained Edge Devices
DOI:
https://doi.org/10.54097/jc1bep53Keywords:
Adaptive inference, constrained reinforcement learning, early exit, edge AI, mixed precision, intent classificationAbstract
Transformer encoders deployed on edge devices must answer each query within a latency deadline while the energy budget, temperature, and memory bandwidth of the device change over time. Hand-designed runtime managers switch among pre-compiled precision modes with feedback rules, but they react to the device rather than to the input and do not plan how a session budget is spent. We cast adaptive inference as a constrained Markov decision process and propose a probe-then-commit controller trained with proximal policy optimization and PID-Lagrangian multipliers. For every query the policy first selects the precision of a three-layer probe, reads the normalized entropy of the probe’s exit classifier, and then either answers immediately or commits a suffix precision and an exit threshold. A feasibility shield masks actions that would violate the response deadline or an energy reserve, and a behavior-cloning warm start from a myopic pacing heuristic stabilizes learning. The controller is trained in a trace-driven environment built from real per-utterance outputs of a RoBERTa-base intent classifier on SLURP and from a batch-1 latency table measured on a CPU core (8.9% MAPE against end-to-end timing). The measurements show that natively supported INT8 and BF16 kernels run 7–15× faster than software-unpacked 4-bit weights and that the layer-9 exit is more accurate than the full network. Across five held-out device scenarios the controller attains 87.00% mean accuracy with no budget depletion, compared with 86.75% for a tuned greedy pacing controller, 86.46% for a tuned re-implementation of a PID runtime manager, and 85.35% for the best static rule.
Downloads
References
[1] Zhou, Z., Chen, X., Li, E., Zeng, L., Luo, K., & Zhang, J. (2019). Edge intelligence: Paving the last mile of artificial intelligence with edge computing. Proceedings of the IEEE, 107(8), 1738–1762. https://doi.org/10.1109/JPROC.2019.2918951
[2] Han, Y., Huang, G., Song, S., Yang, L., Wang, H., & Wang, Y. (2022). Dynamic neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11), 7436–7456. https://doi.org/10.1109/TPAMI.2021.3117837
[3] Teerapittayanon, S., McDanel, B., & Kung, H. T. (2016). BranchyNet: Fast inference via early exiting from deep neural networks. In Proceedings of the 23rd International Conference on Pattern Recognition (pp. 2464–2469). https://doi.org/10.1109/ICPR.2016.7900006
[4] Xin, J., Tang, R., Lee, J., Yu, Y., & Lin, J. (2020). DeeBERT: Dynamic early exiting for accelerating BERT inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 2246–2251). https://doi.org/10.18653/v1/2020.acl-main.204
[5] Liu, W., Zhou, P., Wang, Z., Zhao, Z., Deng, H., & Ju, Q. (2020). FastBERT: A self-distilling BERT with adaptive inference time. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 6035–6044). https://doi.org/10.18653/v1/2020.acl-main.537
[6] Zhou, F. (2026). Moment-constrained side-force allocation for wind-assisted merchant ships: Hydrodynamic performance of retractable anti-leeway fins and flap rudders. Innovation and Technology Studies, 3(1), 27–41.
[7] Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., & Kalenichenko, D. (2018). Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2704–2713). https://doi.org/10.1109/CVPR.2018.00286
[8] Zafrir, O., Boudoukh, G., Izsak, P., & Wasserblat, M. (2019). Q8BERT: Quantized 8bit BERT. In Proceedings of the 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing. https://doi.org/10.1109/EMC2-NIPS53020.2019.00016
[9] Wang, K., Liu, Z., Lin, Y., Lin, J., & Han, S. (2019). HAQ: Hardware-aware automated quantization with mixed precision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8612–8620). https://doi.org/10.1109/CVPR.2019.00881
[10] Shang, W., Teng, D., Chen, T., Zhang, F., & Ding, J. (2026). Neuro-Elastic: A unified framework for hardware-aware adaptive quantization and dynamic sparsity in real-time edge intent prediction. IEEE Access, 14, 111168–111187. https://doi.org/10.1109/ACCESS.2026.3715276
[11] Kaya, Y., Hong, S., & Dumitras, T. (2019). Shallow-deep networks: Understanding and mitigating network overthinking. In Proceedings of the 36th International Conference on Machine Learning (Vol. 97, pp. 3301–3310). PMLR.
[12] Wu, Z., Nagarajan, T., Kumar, A., Rennie, S., Davis, L. S., Grauman, K., & Feris, R. (2018). BlockDrop: Dynamic inference paths in residual networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8817–8826). https://doi.org/10.1109/CVPR.2018.00919
[13] Li, E., Zeng, L., Zhou, Z., & Chen, X. (2020). Edge AI: On-demand accelerating deep neural network inference via edge computing. IEEE Transactions on Wireless Communications, 19(1), 447–457. https://doi.org/10.1109/TWC.2019.2946140
[14] Kim, S., Bin, K., Ha, S., Lee, K., & Chong, S. (2021). zTT: Learning-based DVFS with zero thermal throttling for mobile devices. In Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services. https://doi.org/10.1145/3458864.3468161
[15] Altman, E. (1999). Constrained Markov decision processes. Chapman & Hall/CRC.
[16] Achiam, J., Held, D., Tamar, A., & Abbeel, P. (2017). Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning (Vol. 70, pp. 22–31). PMLR.
[17] Tessler, C., Mankowitz, D. J., & Mannor, S. (2019). Reward constrained policy optimization. In Proceedings of the International Conference on Learning Representations.
[18] Chen, J., Han, X., Zhang, X., & Yang, Y. (2023). Exact concept-level explanations for early detection of financial misstatements. Eurasia Journal of Science and Technology, 5(2), 34–48.
[19] Alshiekh, M., Bloem, R., Ehlers, R., Könighofer, B., Niekum, S., & Topcu, U. (2018). Safe reinforcement learning via shielding. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 32, No. 1, pp. 2669–2678). https://doi.org/10.1609/aaai.v32i1.11797
[20] Huang, S., & Ontañón, S. (2022). A closer look at invalid action masking in policy gradient algorithms. In Proceedings of the International FLAIRS Conference (Vol. 35). [https://doi.org/10.32473/flairs.v35i.130584](https://doi.org/10.32473/flairs.v35i.130584)
[21] Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Osband, I., Dulac-Arnold, G., Agapiou, J., Leibo, J. Z., & Aydemir, A. (2018). Deep Q-learning from demonstrations. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 32, No. 1, pp. 3223–3230). https://doi.org/10.1609/aaai.v32i1.11757
[22] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). RoBERTa: A robustly optimized BERT pretraining approach. https://doi.org/10.48550/arXiv.1907.11692
[23] Yue, X., & Zhu, R. (2024). An explainable hybrid machine learning framework for cash-flow-conditioned multi-horizon liquidity risk early warning: An SME-oriented benchmark study. World Journal of Management Science, 2(1), 63–72.
[24] Skadron, K., Stan, M. R., Huang, W., Velusamy, S., Sankaranarayanan, K., & Tarjan, D. (2003). Temperature-aware microarchitecture. In Proceedings of the 30th Annual International Symposium on Computer Architecture (pp. 2–13). https://doi.org/10.1109/ISCA.2003.1206984
[25] Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2016). High-dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations.
[26] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. https://doi.org/10.48550/arXiv.1707.06347
[27] Dudík, M., Langford, J., & Li, L. (2011). Doubly robust policy evaluation and learning. In Proceedings of the 28th International Conference on Machine Learning (pp. 1097–1104).
[28] Bastianelli, E., Vanzo, A., Swietojanski, P., & Rieser, V. (2020). SLURP: A spoken language understanding resource package. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (pp. 7252–7262). https://doi.org/10.18653/v1/2020.emnlp-main.588
[29] FitzGerald, J., Henchion, M., Unanue, W., Ciaramita, M., Suárez, A., Ahuja, K., Chen, C., Goenaga, D., Mugisha, A., Pruksachatkun, Y., Sitaram, S., Srinivasan, A., Wang, L., & Xia, P. (2023). MASSIVE: A 1M-example multilingual natural language understanding dataset with 51 typologically-diverse languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (pp. 4277–4302). https://doi.org/10.18653/v1/2023.acl-long.235
[30] Larson, S., Mahendran, A., Peper, J. J., Clarke, C., Lee, A., Hill, P., Kummerfeld, J. K., Leach, K., Laurenzano, M. A., Tang, L., & Mars, J. (2019). An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (pp. 1311–1316). https://doi.org/10.18653/v1/D19-1131
[31] Shen, Z., Zhao, W., Wang, B., Wang, Z., & Shang, W. (2026). CAGR: A Cross-Accelerator Graph Optimization Framework for Efficient Recommender System Inference. IEEE Access. https://doi.org/10.1109/ACCESS.2026.3671798
[32] Zhang, H., Ge, Y., Zhao, X., & Wang, J. (2025). Hierarchical Deep Reinforcement Learning for Multi-Objective Integrated Circuit Physical Layout Optimization With Congestion-Aware Reward Shaping. IEEE Access. https://doi.org/10.1109/ACCESS.2025.3610615
Downloads
Published
Issue
Section
License
Copyright (c) 2026 International Journal of Advanced Engineering and Technology Research
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
