Karafan Journal

Karafan Journal

Presenting an "Adaption Ahead" Optimization Algorithm for Training Models Used in Deep Learning

Document Type : Original Article

Authors
Farnaz Hoseini 1
Hamed Sepehrzadeh 1
Masume Kheyri 2
1 Assistant Professor, Department of Computer Engineering, Technical and Vocational University (TVU), Tehran, Iran.
2 The Coach, Department of Computer Engineering, Technical and Vocational University (TVU), Tehran, Iran.
10.48301/kssa.2024.427012.2773
Abstract
Deep learning is a subset of machine learning that is widely used in the field of artificial intelligence such as natural language processing and machine vision. As a subset of machine vision, image segmentation is one of the most common steps in digital image processing, which divides a digital image into different segments. In this research, a new method based on deep learning for image segmentation was presented. The "Adaption Ahead" algorithm was introduced and used as a new optimization algorithm to optimize the proposed model. In previous optimization algorithms, the most important factor in reducing accuracy was extracting low-level features of images and not reducing the semantic distance between human perception and features. In this study, hierarchical and deep feature extraction from images was carried out with the help of deep learning. The "Adaption Ahead" optimization algorithm, in which a deep model based on a convolutional neural network is used, extracted higher-level features and achieved optimal accuracy. By using the Nestrov technique in calculating the gradient by the proposed algorithm, the best result, i.e. 91.1 accuracy, was obtained for the Dice similarity measure. Another advantage of this algorithm over other methods was using uncomplicated calculations. The comparison of the proposed optimization algorithm with other commonly used methods demonstrated the improvement in the performance of this network on relatively large data sets. Furthermore, the more accurate performance of this network, as a result of its hierarchical and deep extraction, was compared to other methods.
Keywords
Deep Learning
Optimization
Optimization Algorithms
Deep Models
Adaption Ahead Algorithm
Subjects
[1] Kwon, H. J., Koo, H. I., & Cho, N. I. (2023). Understanding and explaining convolutional neural networks based on inverse approach. Cognitive Systems Research, 77(1), 142-152. https://doi.org/10.1016/j.cogsys.2022.10.009
[2] Kale, A. P., Wahul, R. M., Patange, A. D., Soman, R., & Ostachowicz, W. (2023). Development of Deep Belief Network for Tool Faults Recognition. Sensors, 23(4), 1872. https://d oi.org/10.3390/s23041872
[3] Dong, S., Wang, P., & Abbas, K. (2021). A survey on deep learning and its applications. Computer Science Review, 40(1), 100379. https://doi.org/10.1016/j.cosrev.2021.100379
[4] Shah, A., Shah, M., Pandya, A., Sushra, R., Sushra, R., Mehta, M., Patel, K., & Patel, K. (2023). A comprehensive study on skin cancer detection using artificial neural network (ANN) and convolutional neural network (CNN). Clinical eHealth, 6(5), 76-84. https://doi. org/10.1016/j.ceh.2023.08.002
[5] He, P., Wang, L., Cui, Y., Wang, R., & Wu, D. (2023). Unsupervised feature learning based on autoencoder for epileptic seizures prediction. Applied Intelligence, 53(18), 20766-20784. https://doi.org/10.1007/s10489-023-04582-9
[6] Bartlett, P. L., Long, P. M., Lugosi, G., & Tsigler, A. (2020). Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(48), 30063-30070. https://doi.org/10.1073/pnas.1907378117
[7] Zheng, Q., Zhu, J., Li, Z., Tian, Z., & Li, C. (2023). Comprehensive Multi-view Representation Learning. Information Fusion, 89(7-8), 198-209. https://doi.org/10.1016/j.inffus.20 22.08.014
[8] Gale, T., Zaharia, M., Young, C., & Elsen, E. (2020, November 09-19). Sparse GPU Kernels for Deep Learning [Conference session]. SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, Atlanta, Georgia, USA. https://doi.org/10.1109/SC41405.2020.00021
[9] Choi, R. Y., Coyner, A. S., Kalpathy-Cramer, J., Chiang, M. F., & Campbell, J. P. (2020). Introduction to Machine Learning, Neural Networks, and Deep Learning. Translational Vision Science & Technology, 9(2), 14-14. https://doi.org/10.1167/tvst.9.2.14
[10] Alizadeh, R., Allen, J. K., & Mistree, F. (2020). Managing computational complexity using surrogate models: a critical review. Research in Engineering Design, 31(3), 275-298. https://doi.org/10.1007/s00163-020-00336-7
[11] Guo, T-D., Liu, Y., & Han, C-Y. (2023). An Overview of Stochastic Quasi-Newton Methods for Large-Scale Machine Learning. Journal of the Operations Research Society of China, 11(2), 245-275. https://doi.org/10.1007/s40305-023-00453-9
[12] Lin, T., Karimireddy, S. P., Stich, S. U., & Jaggi, M. (2021). Quasi-global momentum: Accelerating decentralized deep learning on heterogeneous data . In M. Meila, T. Zhang (Eds.),  Proceedings of the 38th International Conference on Machine Learning (pp. 6654-6665). arXiv. https://doi.org/10.48550/arXiv.2102.04761
[13] Elshamy, R., Abu-Elnasr, O., Elhoseny, M., & Elmougy, S. (2023). Improving the efficiency of RMSProp optimizer by utilizing Nestrove in deep learning. Scientific Reports, 13(1), 8814. https://doi.org/10.1038/s41598-023-35663-x
[14] Huang, F., Wu, X., & Hu, Z. (2023, April 25-27). Adagda: Faster adaptive gradient descent ascent methods for minimax optimization [Conference session]. The 26th International Conference on Artificial Intelligence and Statistics, Valencia, Spain. https://proceed ings.mlr.press/v206/huang23a.html
[15] Huk, M. (2020). Stochastic Optimization of Contextual Neural Networks with RMSprop. In N. T. Nguyen, K. Jearanaitanakij, A. Selamat, B. Trawiński, & S. Chittayasothorn (Eds.), Intelligent Information and Database Systems (pp. 343-352). Springer International Publishing. https://doi.org/10.1007/978-3-030-42058-1_29
[16] Reyad, M., Sarhan, A. M., & Arafa, M. (2023). A modified Adam algorithm for deep neural network optimization. Neural Computing and Applications, 35(23), 17095-17112. https://doi.org/10.1007/s00521-023-08568-z
[17] Daoud, M. S., Shehab, M., Al-Mimi, H. M., Abualigah, L., Zitar, R. A., & Shambour, M. K. Y. (2023). Gradient-Based Optimizer (GBO): A Review, Theory, Variants, and Applications. Archives of Computational Methods in Engineering, 30(4), 2431-2449. https://doi.org/10.1007/s11831-022-09872-y
[18] Wang, M., & Wu, L. (2024, May 07-11). The noise geometry of stochastic gradient descent: A quantitative and analytical characterization [Conference session]. The Twelfth International Conference on Learning Representations, Vienna, Austria. https://doi. org/10.48550/arXiv.2310.00692
[19] Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014, November 03-07). Caffe: Convolutional Architecture for Fast Feature [Conference session]. Embedding. Proceedings of the 22nd Association for Computing Machinery international conference on Multimedia, Orlando, Florida, USA. https://doi.org/10.1145/2647868.2654889
[20] Ceriotti, M., More, J., & Manolopoulos, D. E. (2014). i-PI: A Python interface for ab initio path integral molecular dynamics simulations. Computer Physics Communications, 185(3), 1019-1026. https://doi.org/10.1016/j.cpc.2013.10.027
[21] Hoseini, F., Shahbahrami, A., & Bayat, P. (2019). AdaptAhead Optimization Algorithm for Learning Deep CNN Applied to MRI Segmentation. Journal of Digital Imaging, 32(1), 105-115. https://doi.org/10.1007/s10278-018-0107-6
[22] Yousef, R., Khan, S., Gupta, G., Albahlal, B. M., Alajlan, S. A., & Ali, A. (2023). Bridged-U-Net-ASPP-EVO and Deep Learning Optimization for Brain Tumor Segmentation. Diagnostics, 13(16), 2633. https://doi.org/10.3390/diagnostics13162633
[23] ZainEldin, H., Gamel, S. A., El-Kenawy, E-S. M., Alharbi, A. H., Khafaga, D. S., Ibrahim, A., & Talaat, F. M. (2023). Brain Tumor Detection and Classification Using Deep Learning and Sine-Cosine Fitness Grey Wolf Optimization. Bioengineering, 10(1), 18. https://doi.org/10.3390/bioengineering10010018
[24] Elmezain, M., Mahmoud, A., Mosa, D. T., & Said, W. (2022). Brain Tumor Segmentation Using Deep Capsule Network and Latent-Dynamic Conditional Random Fields. Journal of Imaging, 8(7), 190. https://doi.org/10.3390/jimaging8070190
Karafan Journal
Volume 21, Issue 3
Technical and Engineering
Autumn 2024
Pages 57-83

  • Receive Date 05 December 2023
  • Revise Date 17 January 2024
  • Accept Date 05 April 2024