By default, XGBoost handles these problems by building one stack of scalar-leaf trees for each output. The vector-leaf model offers another choice: build one shared tree whose leaves hold a vector of predictions.

Across ten real-world multiclass datasets, vector leaves generally achieved lower held-out loss and produced simpler models. When outputs benefit from using the same splits, vector-leaf models can also train and predict faster and be less susceptible to spurious features. In this post, we explore when shared tree structure helps and what trade-offs it brings.

Getting started

In XGBoost v3.4.1, the default multi_strategy="one_output_per_tree" builds a separate scalar-leaf tree for each class at every boosting round. Set multi_strategy="multi_output_tree" to build one shared tree with vector-valued leaves instead.

The hist implementation supports CPU and CUDA training, categorical splits, and reduced-gradient training. See the multi-output tutorial for details and examples. Vector leaves were first prototyped for the CPU hist tree method in v2.0; after several releases of refinement, v3.4.1 is the first feature-complete implementation for hist.

This setting changes how the classes share model structure. Each vector leaf stores a separate score update for every class, but all classes follow the same sequence of splits that routes a row to that leaf. For a problem with 32 classes, one scalar boosting round adds 32 trees, while one vector boosting round adds a single shared tree. This difference can produce far fewer splits and a much simpler model.

Why share a tree?

Suppose a tree asks whether a pixel is dark, a playing card is an ace, or a sensor reading exceeds a threshold. In a multiclass problem, the answer may be useful to several classes at once. A vector-leaf tree asks the question once, then stores a different adjustment for every class in the resulting leaf.

This is an appealing example of Occam’s razor: when one partition of the data works as well as many separate partitions, prefer the simpler representation. It can be smaller and faster to evaluate, and the shared structure can act as a useful modeling constraint, making the model more likely to generalize well for unseen data.

The trade-off is that all outputs must use the same tree structure. This works best when they benefit from similar features and split thresholds—a property we call partition compatibility. In multiclass classification, for example, a question about whether a region of an image contains ink may help distinguish several characters at once.

Partition compatibility is not the same as target correlation. Correlation says that target values move together; it does not say that the targets need the same decision boundaries. What matters here is whether the same split questions are useful across outputs. If those questions and thresholds are largely unrelated, separate scalar trees have more freedom.

A synthetic example

To illustrate the benefit of compatible partitions, we generated a multi-output regression problem with 32 normally distributed features. Every target combines those same set of features with different weights and a small amount of random noise. The targets therefore need different predictions, but they all benefit from the same basic split questions. We generated 2^20 training rows and 2^18 test rows, then compared problems with 2, 4, 8, 16, and 32 outputs. Both strategies were trained for 128 boosting rounds. This deliberately partition-compatible dataset is the favorable case for vector leaves.

Both strategies retain essentially the same predictive error. Their complexity, however, separates as the output count grows. At 32 outputs, the vector model is about one ninth of the serialized size, and both training and prediction are faster. Separate scalar trees largely repeat the same useful structure; the vector model represents it once.

Shared partitions resist spurious features

A second controlled experiment tests the regularizing effect of sharing a tree. We generated a synthetic multiclass problem whose class probabilities come from one shared tree over a small set of signal features. Each leaf holds class-specific logits, so the true decision function is exactly partition-compatible.

We then added progressively larger sets of independent noise features. Within each paired comparison, the rows, labels, and signal stay fixed, isolating the effect of giving the models more irrelevant choices. We track how much the added noise worsens expected log loss and, from the tree dumps, how much split gain the models assign to noise.

The curves separate as more distractions become available. At the largest noise setting, the scalar models devote more than twice as much split gain to irrelevant features and suffer more than twice the increase in expected log loss. A scalar class tree can promote a chance association found for one class, whereas a vector tree evaluates each split jointly across all class-score dimensions. Class-specific fluctuations therefore have less influence on the shared partition. Sharing does not eliminate noise splits, but it makes them less competitive.

What happens on real data?

We next compared the strategies on ten multiclass datasets ranging from 13,910 to more than one million rows and from 6 to 1,000 classes. The applications include forest-cover prediction, card hands, motor-drive diagnosis, gas sensing, character recognition, object recognition, and Dionis, a large tabular benchmark with 355 classes from the ChaLearn AutoML Challenge. The suite includes both balanced and heavily imbalanced class distributions.

The figure shows the selected vector model relative to the selected scalar model, with each model selected by validation loss before the test data were evaluated. The dashed line is parity; points to the left favor vector leaves. For example, a split ratio of 0.25 means that the vector model uses one quarter as many splits as the scalar model.

Vector leaves achieve lower held-out loss on nine of ten datasets and use fewer splits and less disk space on all ten. Poker Hand is the standout, but the pattern also appears on balanced datasets such as EMNIST and Devnagari, so class imbalance does not explain the result by itself.

Some gains come with longer training. On Devnagari, the vector model uses about three times as many boosting rounds to train. It nevertheless finishes substantially more accurate and about one third the size. Letter Recognition improves both predictive quality and training time, while Gas Sensor Drift retains a smaller advantage when trained on earlier sensor batches and evaluated on later ones. Shared structure is an advantage when it preserves enough predictive flexibility.

A practical default for multiclass classification

These results make vector leaves a strong starting point for multiclass classification, especially as the number of classes grows. Rather than treating multi_output_tree only as a later optimization, it is reasonable to try it first and retain a scalar model as the comparison.

Compared with scalar models, vector models may need more boosting rounds. Each scalar round adds one tree per class, whereas each vector round adds only one shared tree. An equal round limit therefore gives the scalar model much more tree structure. On K49 and Devnagari, the vector models continued improving for substantially longer and eventually became both more accurate and smaller than their scalar counterparts.

A lower learning rate is also worth pairing with the larger round cap. XGBoost’s default learning rate is quite large. Reducing it makes each tree a smaller update and usually enables more boosting rounds. The smaller steps can make the shared, multidimensional updates easier to optimize. Dionis, for example, improved after its vector learning rate was reduced. Early stopping can then select the useful number of rounds without committing to the full cap.

About the experiments

Every real-data run used GPU histogram training and QuantileDMatrix with a DGX Spark. Validation data controlled model selection and early stopping, and test data were evaluated afterward. Random splits were stratified; Letter Recognition retained its official test partition. Gas Sensor Drift used a time-ordered split because sensor responses change over time. All these experiments use one deterministic split per dataset.

Summary

Vector leaves replace one tree per output with one shared tree whose leaves hold output-specific score updates. When outputs benefit from similar split questions, this can improve predictive quality with far fewer splits and a much smaller model, while reducing the influence of spurious features. The classification results make vector leaves a compelling default to try.

For multiclass problems, start with a generous vector boosting rounds, include a lower learning rate in the validation sweep, and let early stopping select the model. Keep the result when it provides the right balance of predictive quality, model size, and training cost for your application.

XGBoost 3.3.0 is available from the usual package channels. The complete release notes are available in the XGBoost documentation, and release artifacts are available on GitHub.

Faster quantile sketching and distributed training

Quantile sketching received a substantial refactor in 3.3.0. The internals are simpler, the weighted quantile sketch implementation is improved, and GPU quantile sketching uses significantly less memory.

These changes matter because quantile construction is a central part of the histogram-based training pipeline. Reducing memory use and simplifying the implementation improves robustness for large datasets, wide datasets, and GPU workloads where memory pressure often decides whether a job fits.

Distributed quantile construction is also more efficient in this release, using tree reductions to improve scalability. XGBoost no longer requires all-to-all collective connections for distributed training, and users can now customize the worker port when deploying distributed jobs.

Expectile regression

This release adds expectile regression, giving practitioners another tool for risk-sensitive forecasting, uncertainty-aware modeling, and applications where overprediction and underprediction carry different costs. It is available through the new reg:expectileerror objective, the expectile metric, and the expectile_alpha parameter.

Expectile regression is useful when users want an asymmetric squared-loss analogue of quantile regression. It gives a smooth objective for estimating different parts of the conditional response distribution, while retaining a squared-error style penalty.

Multiple expectiles are supported, allowing users to train models for several asymmetric targets in a single configuration. Unlike quantile regression, expectile curves do not suffer from crossing, which makes them easier to use when estimating multiple levels of the response distribution together. This complements XGBoost’s existing regression objectives.

Categorical features are on by default

Categorical feature support is now enabled by default. Users no longer need to set enable_categorical=True in the common case. The parameter remains available for users who need to disable categorical handling explicitly.

This change reflects the maturity of categorical support in XGBoost and makes the Python and dataframe workflows a little more natural. Dataframes with categorical columns can be passed into XGBoost without an extra opt-in step, making the default behavior match what many users already expect from modern tabular machine learning libraries.

The CPU hist tree method also gained one-hot categorical split support for the work-in-progress vector-leaf model. Vector-leaf trees are still under active development, but this release continues to close gaps in the feature set.

SHAP support for vector-leaf trees

XGBoost 3.3.0 expands SHAP support for vector-leaf multi-output trees. Exact SHAP contribution and interaction prediction are now available for vector-leaf multi-output trees on both CPU and GPU.

This is an important step for multi-output models. Vector-leaf trees can model relationships across targets directly by storing a vector of weights in each leaf, but users also need the same interpretability tools they rely on for single-output models. With this release, multi-output workflows gain a more complete explanation story.

The TreeSHAP implementation has also been updated for improved numerical stability and faster execution with QuadratureTreeSHAP [1].

CPU and GPU performance improvements

On CPUs, histogram building has been optimized for wide datasets with column block tiling. XGBoost can also detect CPU cache sizes through Linux sysfs on aarch64, improving its ability to choose efficient execution strategies on modern ARM platforms.

On GPUs, sampling now uses Philox for faster random number generation. This release also continues the wider work on reducing GPU memory pressure and improving the efficiency of the training pipeline.

Training with sampling is now more reproducible within a single session. XGBoost uses a local random number generator and serializes the RNG state in the model configuration, so pickled models can retain that state across save/load boundaries. This helps when training multiple sampled models in the same process.

Python and Spark updates

The minimum supported Python version is now Python 3.12. This lets the project remove older compatibility paths and focus testing on actively supported Python versions.

The Python package also adds PySpark support for Spark Connect ML. This improves compatibility with the newer Spark client-server workflow and keeps the XGBoost Spark integration moving with the broader Spark ecosystem.

The release includes dataframe validation improvements as well. XGBoost now requires Enum support from Polars and validates that pandas column names are unique, helping users catch ambiguous feature inputs earlier.

Several Python fixes and cleanups are included: default verbose behavior is now consistent between XGBClassifier.fit and XGBRegressor.fit, python -OO crashes from missing docstrings have been fixed, boolean indicator features are handled in trees_to_dataframe, and validation messages around feature information and deprecated functions have been improved.

JVM, build, and platform updates

The JVM packages now document Spark 4.0 compatibility. Regressor and ranker pipelines support columnar input, Java packages include automatic module names, and xgboost4j supports FreeBSD.

On the build and platform side, XGBoost 3.3.0 adds support for Visual Studio 2026, updates CUDA Toolkit support, and fixes a number of CMake, macOS, and packaging issues. Linux packaging now uses a versioned shared object.

Deprecation notice

The gblinear booster is deprecated in XGBoost 3.3.0 and will be removed in a future release. The linear model feature in XGBoost has been a niche feature and has not been as actively maintained as the tree algorithms. It also does not support or align with many capabilities in modern XGBoost, such as categorical splits, interaction constraints, monotone constraints, and TreeSHAP-style explanations.

XGBoost is primarily a decision tree library, and we have decided to focus development on the tree algorithms and on keeping the interface consistent for the features users rely on most. Users relying on gblinear should start planning migrations to other estimators or workflows.

Upgrade

For Python users, upgrade with:

pip install --upgrade xgboost

For conda users:

conda install -c conda-forge xgboost

Please see the installation guide for platform-specific instructions, and the 3.3.0 release notes for the full list of changes.

References

[1] Wettenstein R, Mitchell R, Yu P. (2026) Quadrature-TreeSHAP: Depth-Independent TreeSHAP and Shapley Interactions. arXiv:2605.04497.

Many thanks to everyone who contributed code, reviews, testing, documentation, bug reports, and release work for XGBoost 3.3.0.

Histogram based tree construction algorithms

Decision tree construction algorithms typically work by recursively partitioning a set of training instances into smaller and smaller subsets in feature space. These partitions are found by searching over the training instances to find a decision rule that optimises for the training objective. While still effectively linear time these algorithms are slow because searching for the decision rule at the current level requires passing over every training instance. The algorithm can be made considerably faster through discretization of the input features.

Our primary decision tree construction algorithm is now a histogram based method such as that used in [2],[3]. This means that we find quantiles over the input feature space and discretize our training examples into this space. Gradients from the training examples at each boosting iteration can then be summed into histogram ‘bins’ according to the now discrete features. Finding optimal splits for a decision tree then reduces to the simpler problem of searching over histogram bins in a discrete space.

The end result of this is a significantly faster and more memory efficient algorithm that still retains its accuracy.

The above chart shows the difference in execution time on a 1M*50 binary classification problem with 500 boosting iterations.

Multi-GPU support

Our histogram algorithm has full multi-GPU support using the NCCL library for scalable communication between GPUs. This means we can do things like run XGBoost on an AWS P3 instance with eight GPUs. The below chart shows the runtime on the 115M row airline dataset as we increase the number of GPUs:

Because of the efficient AllReduce communication primitives, communication throughput is constant as the number of GPUs is increased. Communication costs are also invariant to the number of training examples because only summary histogram statistics are shared. The data set is evenly distributed between GPUs. This allows us to scale up to datasets that cannot fit on a single GPU and use the full device memory capacity of multi-GPU systems.

Data compression

Our current algorithm is also much more memory efficient than the original algorithm published in [4]. This is achieved largely through data compression of the input matrix after discretization. For example, if we use 256 histogram bins per feature and 50 features, there are only 256*50 unique feature values in the entire input matrix. By using bit compression we can store each matrix element using only log2(256*50)=14 bits per matrix element in a sparse CSR format. For comparison, a naive CSR storage format would typically cost a minimum of 64 bits per matrix element.

The above chart shows the device memory requirements for a 1M*50 binary classification problem on the histogram algorithm and the exact algorithm.

GPU prediction and gradient calculation algorithms

Traditionally, tree construction algorithms account for most of the time spent in a gradient boosting algorithm. This changed after we developed significantly faster tree algorithms and other parts of the gradient boosting process began to create bottlenecks.

Prediction occurs every iteration in gradient boosting in order to calculate the gradients (residuals) for the next iteration. Users may also want to monitor performance on a test or validation set. This adds up to a large amount of computation on the CPU. We map this computation to a GPU kernel for a performance improvement of between 5-10x in prediction time. Note that this improvement is for memory that is already stored on the GPU. When used for an unseen dataset, prediction algorithms will be slower due to the time taken to copy the matrix to the GPU (i.e. these prediction algorithms are designed for training but not deployment of models).

We also introduce GPU accelerated objective function calculation for some tasks. These can be enabled by setting the objective function as one of:

"gpu:reg:linear", "gpu:reg:logistic", "gpu:binary:logistic", gpu:binary:logitraw"

Moving more parts of the gradient boosting pipeline onto the device removes computational bottlenecks as well as reducing the need to copy memory back and forth between the CPU and GPU across the limited bandwidth PCIe bus. Eventually we hope to move the entire pipeline to the device.

Benchmarking against other GBM algorithms

Below are some benchmarks against other gradient boosting algorithms on an 8 GPU cloud computing instance. Instructions for reproducing these benchmarks can be found here.

Installation and usage

From XGBoost version 0.72 onwards, installation with GPU support for python on linux platforms is as simple as:

pip install xgboost

Users of other platforms will still need to build from source, although prebuilt Windows packages are on the roadmap.

To use our new fast algorithms simply set the “tree_method” parameter to “gpu_hist” in your existing XGBoost script.

Simple examples using the XGBoost Python API and sklearn API:

import xgboost as xgb
from sklearn.datasets import load_boston

boston = load_boston()

# XGBoost API example
params = {'tree_method': 'gpu_hist', 'max_depth': 3, 'learning_rate': 0.1}
dtrain = xgb.DMatrix(boston.data, boston.target)
xgb.train(params, dtrain, evals=[(dtrain, "train")])

# sklearn API example
gbm = xgb.XGBRegressor(n_estimators=10, tree_method='gpu_hist')
gbm.fit(boston.data, boston.target, eval_set=[(boston.data, boston.target)])

Output:

[01:12:20] Allocated 0MB on [0] Tesla K80, 11352MB remaining.
[01:12:20] Allocated 0MB on [0] Tesla K80, 11351MB remaining.
[01:12:20] Allocated 0MB on [0] Tesla K80, 11350MB remaining.
[0]     train-rmse:21.6024
[1]     train-rmse:19.5554
[2]     train-rmse:17.7153
[3]     train-rmse:16.0624
[4]     train-rmse:14.5719
[5]     train-rmse:13.2413
[6]     train-rmse:12.0342
[7]     train-rmse:10.9578
[8]     train-rmse:9.97791
[9]     train-rmse:9.10676
[01:12:20] Allocated 0MB on [0] Tesla K80, 11352MB remaining.
[01:12:20] Allocated 0MB on [0] Tesla K80, 11351MB remaining.
[01:12:20] Allocated 0MB on [0] Tesla K80, 11350MB remaining.
[01:12:20] Allocated 0MB on [0] Tesla K80, 11350MB remaining.
[0]     validation_0-rmse:21.6024
[1]     validation_0-rmse:19.5554
[2]     validation_0-rmse:17.7153
[3]     validation_0-rmse:16.0624
[4]     validation_0-rmse:14.5719
[5]     validation_0-rmse:13.2413
[6]     validation_0-rmse:12.0342
[7]     validation_0-rmse:10.9578
[8]     validation_0-rmse:9.97791
[9]     validation_0-rmse:9.10676

Author

Rory Mitchell is a PhD student at the University of Waikato and works for H2O.ai.

Special thanks to all contributors of the XGBoost GPU project, in particular Andrey Adinets and Thejaswi Rao from Nvidia for significant algorithm improvements.

References

[1] Rory Mitchell, Andrey Adinets, Thejaswi Rao: “XGBoost: Scalable GPU Accelerated Learning”, 2018; http://arxiv.org/abs/1806.11248.

[2] Keck, Thomas. “FastBDT: A speed-optimized and cache-friendly implementation of stochastic gradient-boosted decision trees for multivariate classification.” arXiv preprint arXiv:1609.06119 (2016).

[3] Ke, Guolin, et al. “Lightgbm: A highly efficient gradient boosting decision tree.” Advances in Neural Information Processing Systems. 2017.

[4] Mitchell, Rory, and Eibe Frank. “Accelerating the XGBoost algorithm using GPU computing.” PeerJ Computer Science 3 (2017): e127.

Decision tree learning and gradient boosting have until recently been the domain of multicore CPUs. Here we showcase a new plugin providing GPU acceleration for the XGBoost library. The plugin provides significant speedups over multicore CPUs for large datasets.

The plugin can be found at: https://github.com/dmlc/xgboost/tree/master/plugin/updater_gpu

Before talking about the GPU plugin we briefly explain the XGBoost algorithm.

XGBoost for classification and regression

XGBoost is a powerful tool for solving classification and regression problems in a supervised learning setting. It is an implementation of a generalised gradient boosting algorithm designed to offer high-performance, multicore scalability and distributed machine scalability.

The gradient boosting algorithm is an ensemble learning technique that builds many predictive models. Together these smaller models produce much stronger predictions than any single model alone. In particular for gradient boosting, we create these smaller models sequentially, where each new model directly addresses the weaknesses in the previous models.

While many types of models can be used in a boosting algorithm, in practice we almost always use decision trees. Below we show an example of a decision tree that predicts if a person likes computer games based on their age, and gender. Given a new example to predict, we input the example at the root of the tree and follow the decision rules until reaching a leaf node with a prediction.

Given an ensemble of more than one tree we can combine the predictions to obtain a stronger prediction.

It is not uncommon to use XGBoost to create several thousand models such as the above, with each model incrementally improving the result from the previous models.

You may ask why should I care about gradient boosting when machine learning seems to be all about deep learning? The answer is that it works very well for structured data.

XGBoost has become so successful with the Kaggle data science community, to the point of “winning practically every competition in the structured data category”.

GPU Acceleration

Machine learning tasks with XGBoost can take many hours to run. To achieve state-of-the-art prediction results we often want to create thousands of trees and test out many different parameter combinations. It would be nice if users could put their powerful and otherwise idle graphics cards to use accelerating this task.

GPUs launch many thousands of parallel threads at a time and can provide significant speedups for many compute intensive tasks that can be formulated as a parallel algorithm.

Luckily the decision tree construction algorithm may be formulated in parallel, allowing us to accelerate boosting iterations. Note that we parallelise the construction of individual trees - the boosting process itself has a serial dependency.

How fast is it?

The following benchmarks show a performance comparison of GPUs against multicore CPUs for 500 boosting iterations. The new Pascal Titan X shows some nice performance improvements of up to 5.57x as compared to an i7 CPU. The Titan is also able to process the entire 10M row Higgs dataset in its 12GB of memory.

We also tested the Titan X against a server with 2x Xeon E5-2695v2 CPUs (24 cores in total) on the Yahoo learning to rank dataset. The GPU outperforms the CPUs by about 1.2x. This is a nice result considering the Titan X costs $1200 and the 2x Xeon CPUs cost almost $5000.

How does it work?

The XGBoost algorithm requires scanning across gradient/hessian values and using these partial sums to evaluate the quality of splits at every possible split in the training set. The GPU XGBoost algorithm makes use of fast parallel prefix sum operations to scan through all possible splits as well as parallel radix sorting to repartition data. It builds a decision tree for a given boosting iteration one level at a time, processing the entire dataset concurrently on the GPU.

The algorithm also switches between two modes. The first mode processes node groups in interleaved order using specialised multiscan/multireduce operations. This provides better performance at lower levels in the tree when there are fewer leaf nodes. At later levels we switched to using radix sort to repartition the data and perform more conventional scan/reduce operations.

How do I use it?

To use the GPU algorithm add the single parameter:

# Python example
param['updater'] = 'grow_gpu'

XGBoost must be built from source using the cmake build system, following the instructions here.

The plug-in may be used through the Python or CLI interfaces at this time. A demo is available showing how to use the GPU algorithm to accelerate a cross validation task on a large dataset.

About the author

The XGBoost GPU plugin is contributed by Rory Mitchell. The project was a part of a Masters degree dissertation at Waikato University.

Many thanks to the XGBoost authors and contributors!

On March 2016, we released the first version of XGBoost4J, which is a set of packages providing Java/Scala interfaces of XGBoost and the integration with prevalent JVM-based distributed data processing platforms, like Spark/Flink.

The integrations with Spark/Flink, a.k.a. XGBoost4J-Spark and XGBoost-Flink, receive the tremendous positive feedbacks from the community. It enables users to build a unified pipeline, embedding XGBoost into the data processing system based on the widely-deployed frameworks like Spark. The following figure shows the general architecture of such a pipeline with the first version of XGBoost4J-Spark, where the data processing is based on the low-level Resilient Distributed Dataset (RDD) abstraction.

In the last months, we have a lot of communication with the users and gain the deeper understanding of the users’ latest usage scenario and requirements:

• XGBoost keeps gaining more and more deployments in the production environment and the adoption in machine learning competitions Link.

• While Spark is still the mainstream data processing tool in most of scenarios, more and more users are porting their RDD-based Spark programs to DataFrame/Dataset APIs for the well-designed interfaces to manipulate structured data and the significant performance improvement.

• Spark itself has presented a clear roadmap that DataFrame/Dataset would be the base of the latest and future features, e.g. latest version of ML pipeline and Structured Streaming.

Based on these feedbacks from the users, we observe a gap between the original RDD-based XGBoost4J-Spark and the users’ latest usage scenario as well as the future direction of Spark ecosystem. To fill this gap, we start working on the integration of XGBoost and Spark’s DataFrame/Dataset abstraction in September. In this blog, we will introduce the latest version of XGBoost4J-Spark which allows the user to work with DataFrame/Dataset directly and embed XGBoost to Spark’s ML pipeline seamlessly.

A Full Integration of XGBoost and DataFrame/Dataset

The following figure illustrates the new pipeline architecture with the latest XGBoost4J-Spark.

Being different with the previous version, users are able to use both low- and high-level memory abstraction in Spark, i.e. RDD and DataFrame/Dataset. The DataFrame/Dataset abstraction grants the user to manipulate structured datasets and utilize the built-in routines in Spark or User Defined Functions (UDF) to explore the value distribution in columns before they feed data into the machine learning phase in the pipeline. In the following example, the structured sales records can be saved in a JSON file, parsed as DataFrame through Spark’s API and feed to train XGBoost model in two lines of Scala code.

// load sales records saved in json files
val salesDF = spark.read.json("sales.json")
// call XGBoost API to train with the DataFrame-represented training set
val xgboostModel = XGBoost.trainWithDataFrame(
      salesDF, paramMap, numRound, nWorkers, useExternalMemory)

By integrating with DataFrame/Dataset, XGBoost4J-Spark not only enables users to call DataFrame/Dataset APIs directly but also make DataFrame/Dataset-based Spark features available to XGBoost users, e.g. ML Package.

Integration with ML Package

ML package of Spark provides a set of convenient tools for feature extraction/transformation/selection. Additionally, with the model selection tool in ML package, users can select the best model through an automatic parameter searching process which is defined with through ML package APIs. After integrating with DataFrame/Dataset abstraction, these charming features in ML package are also available to XGBoost users.

Feature Extraction/Transformation/Selection

The following example shows a feature transformer which converts the string-typed storeType feature to the numeric storeTypeIndex. The transformed DataFrame is then fed to train XGBoost model.

import org.apache.spark.ml.feature.StringIndexer

// load sales records saved in json files
val salesDF = spark.read.json("sales.json")

// transfrom the string-represented storeType feature to numeric storeTypeIndex
val indexer = new StringIndexer()
  .setInputCol("storeType")
  .setOutputCol("storeTypeIndex")
// drop the extra column
val indexed = indexer.fit(salesDF).transform(df).drop("storeType")

// use the transformed dataframe as training dataset
val xgboostModel = XGBoost.trainWithDataFrame(
      indexed, paramMap, numRound, nWorkers, useExternalMemory)

Pipelining

Spark ML package allows the user to build a complete pipeline from feature extraction/transformation/selection to model training. We integrate XGBoost with ML package and make it feasible to embed XGBoost into such a pipeline seamlessly. The following example shows how to build such a pipeline consisting of feature transformers and the XGBoost estimator.

import org.apache.spark.ml.feature.StringIndexer

// load sales records saved in json files
val salesDF = spark.read.json("sales.json")

// transfrom the string-represented storeType feature to numeric storeTypeIndex
val indexer = new StringIndexer()
  .setInputCol("storeType")
  .setOutputCol("storeTypeIndex")

// assemble the columns in dataframe into a vector
val vectorAssembler = new VectorAssembler()
      .setInputCols(Array("storeId", "storeTypeIndex", ...))
      .setOutputCol("features")

// construct the pipeline       
val pipeline = new Pipeline().setStages(
      Array(storeTypeIndexer, ..., vectorAssembler, new XGBoostEstimator(Map[String, Any]("num_rounds" -> 100)))

// use the transformed dataframe as training dataset
val xgboostModel = pipeline.fit(salesDF)

// predict with the trained model
val salesTestDF = spark.read.json("sales_test.json")
val salesRecordsWithPred = xgboostModel.transform(salesTestDF)

Model Selection

The most critical operation to maximize the power of XGBoost is to select the optimal parameters for the model. Tuning parameters manually is a tedious and labor-consuming process. With the latest version of XGBoost4J-Spark, we can utilize the Spark model selecting tool to automate this process. The following example shows the code snippet utilizing TrainValidationSplit and RegressionEvaluator to search the optimal combination of two XGBoost parameters, [max_depth and eta] (https://github.com/dmlc/xgboost/blob/master/doc/parameter.md). The model producing the minimum cost function value defined by RegressionEvaluator is selected and used to generate the prediction for the test set.

// create XGBoostEstimator 
val xgbEstimator = new XGBoostEstimator(xgboostParam).setFeaturesCol("features").
      setLabelCol("sales")
val paramGrid = new ParamGridBuilder()
      .addGrid(xgbEstimator.maxDepth, Array(5, 6))
      .addGrid(xgbEstimator.eta, Array(0.1, 0.4))
      .build()
val tv = new TrainValidationSplit()
      .setEstimator(xgbEstimator)
      .setEvaluator(new RegressionEvaluator().setLabelCol("sales"))
      .setEstimatorParamMaps(paramGrid)
      .setTrainRatio(0.8)  
val salesTestDF = spark.read.json("sales_test.json")
val salesRecordsWithPred = xgboostModel.transform(salesTestDF)

Summary

Through the latest XGBoost4J-Spark, XGBoost users can build a more efficient data processing pipeline which works with DataFrame/Dataset APIs to handle the structured data with the excellent performance, and simultaneously embrace the powerful XGBoost to explore the insights from the dataset and transform this insight into action. Additionally, XGBoost4J-Spark seamlessly connect XGBoost with Spark ML package which makes the job of feature extraction/transformation/selection and parameter model much easier than before.

The latest version of XGBoost4J-Spark has been available in the [GitHub Repository] (https://github.com/dmlc/xgboost), and the latest API docs are in here.

Portable Machine Learning Systems

XGBoost is one of the projects incubated by Distributed Machine Learning Community (DMLC), which also creates several other popular projects on machine learning systems (Link), e.g. one of the most popular deep learning frameworks, MXNet. We strongly believe that machine learning solution should not be restricted to certain language or certain platform. We realize this design philosophy in several projects, like XGBoost and MXNet. We are willing to see more contributions from the community in this direction.

Further Readings

If you are interested in knowing more about XGBoost, you can find rich resources in

• The github repository of XGBoost
• The comprehensive documentation site for XGBoostl
• An introduction to the gradient boosting model
• Tutorials for the R package
• Introduction of the Parameters
• Awesome XGBoost, a curated list of examples, tutorials, blogs about XGBoost usecases

XGBoost is a library designed and optimized for tree boosting. Gradient boosting trees model is originally proposed by Friedman et al. By embracing multi-threads and introducing regularization, XGBoost delivers higher computational power and more accurate prediction. More than half of the winning solutions in machine learning challenges hosted at Kaggle adopt XGBoost (Incomplete list). XGBoost has provided native interfaces for C++, R, python, Julia and Java users. It is used by both data exploration and production scenarios to solve real world machine learning problems.

The distributed XGBoost is described in the recently published paper. In short, the XGBoost system runs magnitudes faster than existing alternatives of distributed ML, and uses far fewer resources. The reader is more than welcomed to refer to the paper for more details.

Despite the current great success, one of our ultimate goals is to make XGBoost even more available for all production scenario. Programming languages and data processing/storage systems based on Java Virtual Machine (JVM) play the significant roles in the BigData ecosystem. Hadoop, Spark and more recently introduced Flink are very useful solutions to general large-scale data processing.

On the other side, the emerging demands of machine learning and deep learning inspires many excellent machine learning libraries. Many of these machine learning libraries(e.g. XGBoost/MxNet) requires new computation abstraction and native support (e.g. C++ for GPU computing). They are also often much more efficient.

The gap between the implementation fundamentals of the general data processing frameworks and the more specific machine learning libraries/systems prohibits the smooth connection between these two types of systems, thus brings unnecessary inconvenience to the end user. The common workflow to the user is to utilize the systems like Spark/Flink to preprocess/clean data, pass the results to machine learning systems like XGBoost/MxNet) via the file systems and then conduct the following machine learning phase. This process jumping across two types of systems creates certain inconvenience for the users and brings additional overhead to the operators of the infrastructure.

We want best of both worlds, so we can use the data processing frameworks like Spark and Flink toghether with the best distributed machine learning solutions. To resolve the situation, we introduce the new-brewed XGBoost4J, XGBoost for JVM Platform. We aim to provide the clean Java/Scala APIs and the integration with the most popular data processing systems developed in JVM-based languages.

Unix Philosophy in Machine Learning

XGBoost and XGBoost4J adopts Unix Philosophy. XGBoost does its best in one thing – tree boosting and is being designed to work with other systems. We strongly believe that machine learning solution should not be restricted to certain language or certain platform.

Specifically, users will be able to use distributed XGBoost in both Spark and Flink, and possibly more frameworks in Future. We have made the API in a portable way so it can be easily ported to other Dataflow frameworks provided by the Cloud. XGBoost4J shares its core with other XGBoost libraries, which means data scientists can use R/python read and visualize the model trained distributedly. It also means that user can start with single machine version for exploration, which already can handle hundreds of million examples.

System Overview

In the following Figure, we describe the overall architecture of XGBoost4J. XGBoost4J provides the Java/Scala API calling the core functionality of XGBoost library. Most importantly, it not only supports the single-machine model training, but also provides an abstraction layer which masks the difference of the underlying data processing engines and scales training to the distributed servers.

By calling the XGBoost4J API, users can scale the model training to the cluster. XGBoost4J calls the running instance of XGBoost worker in Spark/Flink task and run them across the cluster. The communication among the distributed model training tasks and the XGBoost4J runtime environment go through [Rabit] (https://github.com/dmlc/rabit).

With the abstraction of XGBoost4J, users can build an unified data analytic application ranging from Extract-Transform-Loading, data exploration, machine learning model training and the final data product service. The following figure illustrate an example application built on top of Apache Spark. The application seamlessly embeds XGBoost into the processing pipeline and exchange data with other Spark-based processing phase through Spark’s distributed memory layer.

Single-machine Training Walk-through

In this section, we will work through the APIs of XGBoost4J by examples. We will be using scala for demonstration, but we also have a complete API for java users.

To start the model training and evaluation, we need to prepare the training and test set:

val trainMax = new DMatrix("../../demo/data/agaricus.txt.train")
val testMax = new DMatrix("../../demo/data/agaricus.txt.test")

After preparing the data, we can train our model:

val params = new mutable.HashMap[String, Any]()
params += "eta" -> 1.0
params += "max_depth" -> 2
params += "objective" -> "binary:logistic"

val watches = new mutable.HashMap[String, DMatrix]
watches += "train" -> trainMax
watches += "test" -> testMax

val round = 2
// train a model
val booster = XGBoost.train(trainMax, params.toMap, round, watches.toMap)

We then evaluate our model:

val predicts = booster.predict(testMax)

predict can output the predict results and you can define a customized evaluation method to derive your own metrics (see the example in (Customized Evaluation Metric in Java, [Customized Evaluation Metric in Scala] (https://github.com/dmlc/xgboost/blob/master/jvm-packages/xgboost4j-example/src/main/scala/ml/dmlc/xgboost4j/scala/example/CustomObjective.scala)).

Distributed Model Training with Distributed Dataflow Frameworks

The most exciting part in this XGBoost4J release is the integration with the Distributed Dataflow Framework. The most popular data processing frameworks fall into this category, e.g. Apache Spark, [Apache Flink] (http://flink.apache.org/), etc. In this part, we will walk through the steps to build the unified data analytic applications containing data preprocessing and distributed model training with Spark and Flink. (currently, we only provide Scala API for the integration with Spark and Flink)

Similar to the single-machine training, we need to prepare the training and test dataset.

Spark Example

In Spark, the dataset is represented as the Resilient Distributed Dataset (RDD), we can utilize the Spark-distributed tools to parse libSVM file and wrap it as the RDD:

val trainRDD = MLUtils.loadLibSVMFile(sc, inputTrainPath).repartition(args(1).toInt)

We move forward to train the models:

val xgboostModel = XGBoost.train(trainRDD, paramMap, numRound, numWorkers)

The next step is to evaluate the model, you can either predict in local side or in a distributed fashion

// testSet is an RDD containing testset data represented as
// org.apache.spark.mllib.regression.LabeledPoint
val testSet = MLUtils.loadLibSVMFile(sc, inputTestPath)

// local prediction
// import methods in DataUtils to convert Iterator[org.apache.spark.mllib.regression.LabeledPoint]
// to Iterator[ml.dmlc.xgboost4j.LabeledPoint] in automatic
import DataUtils._
xgboostModel.predict(new DMatrix(testSet.collect().iterator)

// distributed prediction
xgboostModel.predict(testSet)

Flink example

In Flink, we represent training data as Flink’s DataSet

val trainData = MLUtils.readLibSVM(env, "/path/to/data/agaricus.txt.train")

Model Training can be done as follows

val xgboostModel = XGBoost.train(trainData, paramMap, round)

Training and prediction.

// testData is a Dataset containing testset data represented as
// org.apache.flink.ml.math.Vector.LabeledVector
val testData = MLUtils.readLibSVM(env, "/path/to/data/agaricus.txt.test")

// local prediction
xgboostModel.predict(testData.collect().iterator)

// distributed prediction
xgboostModel.predict(testData.map{x => x.vector})

Road Map

It is the first release of XGBoost4J package, we are actively move forward for more charming features in the next release. You can watch our progress in XGBoost4J Road Map.

While we are trying our best to keep the minimum changes to the APIs, it is still subject to the incompatible changes.

Further Readings

If you are interested in knowing more about XGBoost, you can find rich resources in

• The github repository of XGBoost
• The comprehensive documentation site for XGBoostl
• An introduction to the gradient boosting model
• Tutorials for the R package
• Introduction of the Parameters
• Awesome XGBoost, a curated list of examples, tutorials, blogs about XGBoost usecases

Acknowledgements

We would like to send many thanks to Zixuan Huang, the early developer of XGBoost for Java (XGBoost for Java).

XGBoost is a library designed and optimized for boosting trees algorithms. Gradient boosting trees model is originally proposed by Friedman et al. The underlying algorithm of XGBoost is similar, specifically it is an extension of the classic gbm algorithm. By employing multi-threads and imposing regularization, XGBoost is able to utilize more computational power and get more accurate prediction. Please refer to this tutorial for the details of the model.

One evidence of its accuracy is that XGBoost is used in more than half of the winning solutions in machine learning challenges hosted at Kaggle. We have prepared a (incomplete) list of winning solutions.

There’re various high-level interfaces. Currently there are interfaces of XGBoost in C++, R, python, Julia, Java and Scala. The core functions in XGBoost are implemented in C++, thus it is easy to share models among different interfaces. This post is going to focus on the R package xgboost, which has a friendly user interface and comprehensive documentation. Based on the statistics from the RStudio CRAN mirror, The package has been downloaded for more than 4,000 times in the last month.

The R package xgboost has won the 2016 John M. Chambers Statistical Software Award. From the very beginning of the work, our goal is to make a package which brings convenience and joy to the users. Thus we will introduce several details of the R pacakge xgboost that (we think) users would love to know.

A 1-minute Beginner’s Guide

xgboost is available on both CRAN and Github. To install the stable/pre-compiled version from CRAN, simply run:

install.packages('xgboost')

You can also install from our weekly updated drat repo:

install.packages("drat", repos="https://cran.rstudio.com")
drat:::addRepo("dmlc")
install.packages("xgboost", repos="http://dmlc.ml/drat/", type="source")

In order to run a machine learning algorithm, we need a data set first. xgboost has a demo data set about mushrooms. This data set records biological attributes of different mushroom species, and the target is to predict whether it is poisonous. Users can load the data with

require(xgboost)

data(agaricus.train, package='xgboost')
data(agaricus.test, package='xgboost')
train <- agaricus.train
test <- agaricus.test

Each variable is a list containing two things, label and data. The next step is to feed this data to xgboost. Besides the data, we need to train the model with some other parameters:

• nrounds: the number of decision trees in the final model
• objective: the training objective to use, where “binary:logistic” means a binary classifier.

The simplest training command is as follows:

model <- xgboost(data = train$data, label = train$label,
                 nrounds = 2, objective = "binary:logistic")

## [0]	train-error:0.000614
## [1]	train-error:0.001228

We can make prediction on the test data set easily:

preds = predict(model, test$data)

Sometimes it is important to use cross-validation to exam our model. In xgboost we provide a function xgb.cv to do that. Basically users can just copy every thing from xgboost, and specify nfold:

cv.res <- xgb.cv(data = train$data, label = train$label, nfold = 5,
                 nrounds = 2, objective = "binary:logistic")

## [0]	train-error:0.000921+0.000343	test-error:0.001228+0.000687
## [1]	train-error:0.001075+0.000172	test-error:0.001228+0.000687

To get to know more details about the usage of xgboost, please visit our tutorial for more information.

Efficient Algorithm

If you have experiences in training models on a large data set, then you probably agree that waiting for the training to be done is boring. Time is a resource, so the training speed of an learning algorithm is important. This is determined by both algorithm and implementation. We pay attention to these issues when building xgboost, thus we are confident that xgboost is one of the fastest learning algorithm of gradient boosting algorithm. The reasons for the good efficiency are:

• The computational part is implemented in C++.
• It can be multi-threaded on a single machine.
• It preprocesses the data before the training algorithm.

This figure is generated with the dataset from the Higgs Boson Competition. It can be described from two aspects:

• With only one thread, the effect of preprocessing and C++ is already obvious.
• The multi-threading is almost linear with the number of threads, thus boosting the efficiency further.

Convenient Interface

As the developers of xgboost, we are also heavy users of xgboost. We value the experience on this tool. During the development, we try to shape the package to be user-friendly. Here are several details we would like to share, please click the title to visit the sample code.

Customized Objective

xgboost can take customized objective. This means the model could be trained to optimize the objective defined by user. This is not often seen in other tools, since most of the algorithms are binded with a specific objective. With xgboost, one can train a model which maximize the work on the correct direction.

Let’s use log-likelihood as an demo. We define the following function to calculate the first and second order of gradient of the loss function:

loglossobj <- function(preds, dtrain) {
  # dtrain is the internal format of the training data
  # We extract the labels from the training data
  labels <- getinfo(dtrain, "label")
  # We compute the 1st and 2nd gradient, as grad and hess
  preds <- 1/(1 + exp(-preds))
  grad <- preds - labels
  hess <- preds * (1 - preds)
  # Return the result as a list
  return(list(grad = grad, hess = hess))
}

Then we train the model as

model <- xgboost(data = train$data, label = train$label,
                 nrounds = 2, objective = loglossobj, eval_metric = "error")

## [0]	train-error:0.001228
## [1]	train-error:0.001228

The result should be equivalent to objective = "binary:logistic".

Early Stopping

A usual scenario is when we are not sure how many trees we need, we will firstly try some numbers and check the result. If the number we try is too small, we need to make it larger; If the number is too large, we are wasting time to wait for the termination. By setting the parameter early_stopping, xgboost will terminate the training process if the performance is getting worse in the iteration.

bst <- xgb.cv(data = train$data, label = train$label, nfold = 5,
              nrounds = 20, objective = "binary:logistic",
              early.stop.round = 3, maximize = FALSE)

## [0]	train-error:0.000921+0.000343	test-error:0.001228+0.000686
## [1]	train-error:0.001228+0.000172	test-error:0.001228+0.000686
## [2]	train-error:0.000653+0.000442	test-error:0.001075+0.000875
## [3]	train-error:0.000422+0.000416	test-error:0.000767+0.000940
## [4]	train-error:0.000192+0.000429	test-error:0.000460+0.001029
## [5]	train-error:0.000192+0.000429	test-error:0.000460+0.001029
## [6]	train-error:0.000000+0.000000	test-error:0.000000+0.000000
## [7]	train-error:0.000000+0.000000	test-error:0.000000+0.000000
## [8]	train-error:0.000000+0.000000	test-error:0.000000+0.000000
## [9]	train-error:0.000000+0.000000	test-error:0.000000+0.000000
## Stopping. Best iteration: 7

Here we are doing cross validation. early.stop.round = 3 means if the performance is not getting better for 3 steps, then the program will stop. maximize = FALSE means our goal is not to maximize the evaluation, where the default evaluation metric for binary classification is the classification error rate. We can see that even we ask the model to train 20 trees, it stopped after the performance is perfect.

Continue Training

Sometimes we might want to try to do 1000 iterations and check the result, then decide if we need another 1000 ones. Usually the second step could only be done by starting from the beginning, again. In xgboost users can continue the training on the previous model, thus the second step will cost you the time for the additional iterations only. The theoratical reason that we are capable to do this is because each tree is only trained based the prediction result of the previous trees. Once we get the prediction by the current trees, we can start to train the next one.

This feature involves with the internal data format of xgboost: xgb.DMatrix. An xgb.DMatrix object contains the features, target and other side informations, e.g. weights, missing values.

First, let us define an xgb.DMatrix object for this data:

dtrain <- xgb.DMatrix(train$data, label = train$label)

Next we train the model with it:

model <- xgboost(data = dtrain, nrounds = 2, objective = "binary:logistic")

## [0]	train-error:0.000614
## [1]	train-error:0.001228

Note that we have the label included in the dtrain object. Then we make prediction on the current training data:

pred_train <- predict(model, dtrain, outputmargin=TRUE)

Here the parameter outputmargin indicates that we don’t need a logistic transformation of the result.

Finally we put the previous prediction result as an additional information to the object dtrain, so that the training algorithm knows where to start.

setinfo(dtrain, "base_margin", pred_train)

## [1] TRUE

Now observe how is the result changed:

model <- xgboost(data = dtrain, nrounds = 2, objective = "binary:logistic")

## [0]	train-error:0.000614
## [1]	train-error:0.000614

Handle Missing Values

Missing value is commonly seen in real-world data sets. Handling missing values has no rule to apply to all cases, since there could be various reasons for the values to be missing. In xgboost we choose a soft way to handle missing values. When using a feature with missing values to do splitting, xgboost will assign a direction to the missing values instead of a numerical value. Specifically, xgboost guides all the data points with missing values to the left and right respectively, then choose the direction with a higher gain with regard to the objective.

To enable this feature, simply set the parameter missing to mark the missing value label. To demonstrate it, we can manually make a dataset with missing values.

dat <- matrix(rnorm(128), 64, 2)
label <- sample(0:1, nrow(dat), replace = TRUE)
for (i in 1:nrow(dat)) {
  ind <- sample(2, 1)
  dat[i, ind] <- NA
}

Then we only need to specify the missing value marker, NA, in our code:

model <- xgboost(data = dat, label = label, missing = NA,
                 nrounds = 2, objective = "binary:logistic")

## [0]	train-error:0.281250
## [1]	train-error:0.281250

Practically, the default value of missing is exactly NA, therefore we don’t even need to specify it in a standard case.

Model Inspection

The model used by xgboost is gradient boosting trees, therefore a model usually contains multiple tree models. A typical ensemble of two trees looks like this:

bst <- xgboost(data = train$data, label = train$label, max.depth = 2,
               eta = 1, nthread = 2, nround = 2, objective = "binary:logistic")
xgb.plot.tree(feature_names = agaricus.train$data@Dimnames[[2]], model = bst)

We can make an interpretation on the model easily. xgboost provides a function xgb.plot.tree to plot the model so that we can have a direct impression on the result.

However, what if we have way more trees?

bst <- xgboost(data = train$data, label = train$label, max.depth = 2,
               eta = 1, nthread = 2, nround = 10, objective = "binary:logistic")
xgb.plot.tree(feature_names = agaricus.train$data@Dimnames[[2]], model = bst)

It is starting to make things messy. We even have a hard time to inspect every detail on the plot. It is not easy to tell a story with too many conditions.

Multiple-in-one plot

In xgboost, we provide a function xgb.plot.multi.trees to ensemble several trees into a single one! This function is inspired by this blogpost: https://wellecks.wordpress.com/2015/02/21/peering-into-the-black-box-visualizing-lambdamart/. This is done with the following observations:

• Almost all the trees in an ensemble model have the same shape. If the maximum depth is determined, this holds for all the binary trees.
• On each node there would be more than one feature that have appeared on this position. But we can describe it by the frequency of each feature thus make a frequenct table.

Here is an example of an “ensembled” tree visualization.

bst <- xgboost(data = train$data, label = train$label, max.depth = 15,
                 eta = 1, nthread = 2, nround = 30, objective = "binary:logistic",
                 min_child_weight = 50)
xgb.plot.multi.trees(model = bst, feature_names = agaricus.train$data@Dimnames[[2]], features.keep = 3)

The text in the nodes indicates the distribution of the features selected at this position. If we hover our mouse on the nodes, we get hte information of the path.

Feature Importance

If the tree is too deep, or the number of features is large, then it is still gonna be difficult to find any useful patterns. One simplified way is to check feature importance instead. How do we define feature importance in xgboost?

In xgboost, each split tries to find the best feature and splitting point to optimize the objective. We can calculate the gain on each node, and it is the contribution from the selected feature. In the end we look into all the trees, and sum up all the contribution for each feature and treat it as the importance. If the number of features is large, we can also do a clustering on features before we make the plot. Here’s an example of the feature importance plot from the function xgb.plot.importance:

bst <- xgboost(data = train$data, label = train$label, max.depth = 2,
               eta = 1, nthread = 2, nround = 2,objective = "binary:logistic")
importance_matrix <- xgb.importance(agaricus.train$data@Dimnames[[2]], model = bst)
xgb.plot.importance(importance_matrix)

Deepness

There is more than one way to understand the structure of the trees, besides plotting them all. Since there are all binary trees, we can have a clear figure in mind if we get to know the depth of each leaf. The function xgb.plot.deepness is inspired by this blogpost: http://aysent.github.io/2015/11/08/random-forest-leaf-visualization.html.

From the function xgb.plot.deepness, we can get two plots summarizing the distribution of leaves according to the change of depth in the tree.

bst <- xgboost(data = train$data, label = train$label, max.depth = 15,
                 eta = 1, nthread = 2, nround = 30, objective = "binary:logistic",
                 min_child_weight = 50)
xgb.plot.deepness(model = bst)

The upper plot shows the number of leaves per level of deepness. The lower plot shows noramlized weighted cover per leaf (weighted sum of instances). From this information, we can see that for the 5-th and 6-th level, there are actually not many leaves. To avoid overfitting, we can restrict the depth of trees to be a small number.

Further Readings

This blog post only brings you a glimpse on XGBoost, while there are a lot more exciting resources about it! Feel free to go through the list and look for whatever you like.

• The github repository of XGBoost
• The comprehensive documentation site for XGBoostl
• An introduction to the gradient boosting model
• Tutorials for the R package
• Introduction of the Parameters
• Awesome XGBoost, a curated list of examples, tutorials, blogs about XGBoost usecases

If there is any question, please feel free to check out the issue forum.