Comparison of Non-Convex Variable Selection Criteria in High-dimensional Data with Count Response
Background: High-dimensional datasets with count responses present unique challenges in variable selection due to the large number of predictors and potential collinearity. While significant work has been done on selecting predictors from high-dimensional data with metrical covariates and Gaussian responses, there is a gap in comparing the performances of SCAD (Smoothly Clipped Absolute Deviation) and MCP (Minimax Concave Penalty) across different tuning parameters for count responses. Such comparisons, especially with k-fold cross-validation, are rare, highlighting the need for this study. Aim: This study aims to compare the effectiveness of SCAD and MCP in high-dimensional datasets where the response variable is a count, specifically for various tuning parameters and cross-validation techniques. Method: A simulation study was conducted using datasets with 250 predictors and a sample size of 100 modelled by a Poisson distribution (without overdispersion). The performance of SCAD and MCP was evaluated based on selection accuracy, cross-validation error, and computational efficiency. Results: The results showed that SCAD's performance at tested tuning parameters was similar to the standard tuning parameter (γ = 3.7) at 10-fold cross-validation only. For MCP, the variable selection performance at tuning parameters γ = 4 and γ = 5 was comparable to the standard (γ = 3), while other values differed significantly. Additionally, 10-fold cross-validation yielded the lowest cross-validation errors, and MCP selected the fewest predictors. Conclusion: Non-convex variable selection methods, particularly SCAD and MCP, are recommended for their robustness and accuracy in identifying relevant predictors and enhancing predictive capabilities in high-dimensional datasets with count responses.
Download Paper