This study benchmarks CatBoost, XGBoost, and TabPFN V2 across multiple Kaggle datasets, including classification, regression, and time series tasks.TabPFN V2 showed strong performance on small datasets, often outperforming the boosting models without any feature engineering or preprocessing. However, it struggles with large datasets over 10,000 rows due to memory limitations, where XGBoost remains more effective, particularly for time series forecasting. Overall, TabPFN V2 is a powerful tool for quick experimentation on smaller tabular datasets but is less practical for large-scale applications.The objective of this study is to benchmark the performance of three models — CatBoost,XGBoost, and TabPFN — across multiple Kaggle datasets. The aim is to evaluate how thesemodels perform on diverse tasks including binary classification, regression, and time seriesforecasting. The focus is on comparing predictive accuracy, error metrics, and generalapplicability of TabPFN (a transformer-based model designed for tabular data) againstwell-established gradient boosting models.The experiments were conducted on four Kaggle datasets covering different predictive modelingtasks. A summary of each dataset is provided below:Dataset NameTask TypeTarget VariableNumber of SamplesNumber of FeaturesNotesTitanic - Machine Learning from DisasterBinary ClassificationSurvived (0/1)891 (train set)11 (after cleaning)Predict passenger survivalHouse Prices - Advanced Regression TechniquesRegressionSalePrice1,46079Predict house sale pricesBinary Prediction with a Rainfall DatasetBinary ClassificationRainTomorrow (Yes/No)2,19012Predict if it will rain tomorrowForecasting Sticker SalesTime Series Forecastingnum_sold230,1305 (date, country, store, product, id)Predict sticker sales; missing values present in num_soldThe models were evaluated based on metrics appropriate to each task. For the House Pricesdataset, RMSLE (Root Mean Squared Log Error) was used in accordance with Kaggle’s officialevaluation methodDataset NameMetricCatBoostXGBoostTabPFNNotesTitanic - Machine Learning from DisasterAccuracy0.775110.735840.78947House Prices - Advanced Regression TechniquesRMSLE (Root Mean Squared Log Error)0.12830.155540.11359Evaluated on log(predictions)vslog(actual) as per KaggleBinary Prediction with a Rainfall DatasetAUC ROC0.842310.84580.86564Forecasting Sticker SalesRMSE102.0889.5699.24All models were trained on a sampled dataset of 10,000 entries.During this benchmarking study, the following limitations were encountered:This benchmarking study demonstrates that TabPFN, trained on large-scale synthetic data andleveraging in-context learning, can outperform or match established models like CatBoost andXGBoost on small datasets with minimal preprocessing. Its ability to handle raw tabular datawithout requiring feature engineering makes it a powerful tool for quick experimentation andbaseline modeling.However, significant limitations arise when working with larger datasets beyond 10,000 rows, asnoted by the model’s creators. Attempts to run TabPFN on large datasets resulted in memoryoverloads and kernel crashes, making it impractical for large-scale training.In summary:Every data science course introduces Principal Component Analysis. But most explanations stop at “how it works” - and rarely answer the more practical question: “When should you actually reach for it? When will it quietly hurt you?” This post is about that gap.

Machine learning teams love clean numbers: accuracy of 92 percent. RMSE of 3.4, F1 score of 0.88. These numbers feel objective, precise, and reassuring. They fit neatly into dashboards, slide decks, and go or no-go decisions.Abdukarimov HumoyunOn this page© 2026 HumbleBee - notes on AI, engineering, and the work itself.