Updates

• Add some advanced off-policy gradient estimators (#167)
• Automatic candidate hyperparamer sorting for slope (#168)
• Fixing the the error checking about "p_e_a" in obp.ope.OffPolicyEvaluation (#169)
• Fixing the expected reward factual in the independent reward structure (#170)
• Allowing slope to use the true marginal importance weight for mips (#172)

References

• Yuta Saito and Thorsten Joachims. "Off-Policy Evaluation for Large Action Spaces via Embeddings." 2022.
• Thorsten Joachims, Adith Swaminathan, and Maarten de Rijke. "Deep Learning for Logged Bandit Feedback.", 2018.
• Yi Su, Maria Dimakopoulou, Akshay Krishnamurthy, and Miroslav Dudik.Doubly Robust Off-Policy Evaluation with Shrinkage.", 2020.
• Alberto Maria Metelli, Alessio Russo, and Marcello Restelli. "Subgaussian and Differentiable Importance Sampling for Off-Policy Evaluation and Learning.", 2021.

Updates

• Implement a synthetic data generator class for OPE with action embeddings (obp.dataset.SyntheticBanditDatasetWithActionEmbeds) and an estimator leveraging the action embeddings (obp.ope.MarginalziedInverseProbabilityWeighting) (#155 )
• Implement several OPE estimators for the multiple logger setting (#154 )

References

• Yuta Saito and Thorsten Joachims. "Off-Policy Evaluation for Large Action Spaces via Embeddings." arXiv2022.
• Aman Agarwal, Soumya Basu, Tobias Schnabel, Thorsten Joachims. "Effective Evaluation using Logged Bandit Feedback from Multiple Loggers.", KDD2018.
• Nathan Kallus, Yuta Saito, and Masatoshi Uehara. "Optimal Off-Policy Evaluation from Multiple Logging Policies.", ICML2021.

Updates

• Implement obp.policy.QLearner (#144 )
• Implement Balanced IPW Estimator as obp.ope.BalancedInverseProbabilityWeighting. See Sondhi et al.(2020) for details. (#146 ).
• Implement the Cascade Doubly Robust estimator for the combinatorial action OPE as obp.ope.CascadeDR. See Kiyohara et al.(2022) for details. (#142 )
• Implement a data-driven hyperparameter tuning method for OPE called SLOPE proposed by Su et al.(2020) and Tucker et al.(2021) (#148 )
• Implement new estimators for the standard OPE based on a power-mean transformation of importance weights proposed by Metelli et al.(2021) (#149 )
• Implement dataset class for generating synthetic logged bandit data with multiple loggers. Corresponding estimators will be added in the next update (#150 )
• Implement an argument to control the number of deficient actions in obp.dataset.SyntheticBanditDataset and obp.dataset.MultiClassToBanditReduction. See Sachdeva et al.(2020) for details. (#150 )
• Implement some flexible functions to synthesize reward function and behavior policy (#145 )

Minors

• Adjust to sklearn>=1.0.0
• Fix/update error messages, docstrings, and examples.

References

• Haruka Kiyohara, Yuta Saito, Tatsuya Matsuhiro, Yusuke Narita, Nobuyuki Shimizu, and Yasuo Yamamoto. Doubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Model. WSDM2022.
• Arjun Sondhi, David Arbour, Drew Dimmery. Balanced Off-Policy Evaluation in General Action Spaces. AISTATS2020.
• Yi Su, Pavithra Srinath, Akshay Krishnamurthy. Adaptive Estimator Selection for Off-Policy Evaluation. ICML2020.
• George Tucker and Jonathan Lee. Improved Estimator Selection for Off-Policy Evaluation. 2021
• Alberto Maria Metelli, Alessio Russo, Marcello Restelli. Subgaussian and Differentiable Importance Sampling for Off-Policy Evaluation and Learning. NeurIPS2021.
• Noveen Sachdeva, Yi Su, and Thorsten Joachims. "Off-policy Bandits with Deficient Support.", KDD2020.
• Aman Agarwal, Soumya Basu, Tobias Schnabel, Thorsten Joachims. "Effective Evaluation using Logged Bandit Feedback from Multiple Loggers.", KDD2018.
• Nathan Kallus, Yuta Saito, and Masatoshi Uehara. "Optimal Off-Policy Evaluation from Multiple Logging Policies.", ICML2021.

The changes are summarized below:

Major updates

• Add OPE/OPE with Continuous Actions
  • SyntheticContinuousBanditDataset (#112 )
  • ContinuousOPEEstimators [1] (#113 )
  • ContinuousNNPolicyLearner [2] (#114 )
• Add Weight clipping to IPW and DR (#115 )
• Add Automatic Hyperparameter Tuning of OPE estimators [3] (#116, #131 )
• Add arguments to the SyntheticBanditDataset class to generate more flexible synthetic data (#123 )
• Add subsample option to the OpenBanditDataset class (#118 )
• Modify an input type of off_policy_objective argument and Add some hyperparameters to NNPolicyLearner (#132)

Minor updates

• Fix README (#119 )
• Fix Scalar value checking (#122 )
• Add ValueError to OffPolicyEvaluation class (#125 )
• Fix Error messages (#126 )
• Add Some Errors (#125, #129 )
• Update Quickstart examples (#127 )

Cautions

• the hyperparameter name of obp.ope.SwitchDoublyRobust has changed to lambda_ from tau
• the type of argument off_policy_objective of obp.policy.NNPolicyLearner has changed to str from callable

References

• Nathan Kallus and Angela Zhou. Policy Evaluation and Optimization with Continuous Treatments, AISTATS2018.
• Nathan Kallus and Masatoshi Uehara. "Doubly Robust Off-Policy Value and Gradient Estimation for Deterministic Policies", NeurIPS2020.
• Yi Su, Maria Dimakopoulou, Akshay Krishnamurthy, and Miroslav Dudik. "Doubly Robust Off-Policy Evaluation with Shrinkage.", ICML2020.

The changes are summarized below:

• Add some functions to implement OPE for slate contextual bandit setting [1]

  • SlateSyntheticBanditFeedback (#82, #93, #95, #98, #100, #101, #102, #104, #105)
  • Slate OPE Estimators (#88)

• Make OffPolicyEvaluation class more useful

  • add a method to visualize and compare OPE results of several different policies (#103)
  • Enable to use different estimated_rewards_by_reg_model values (this will make MRDR [2] easier to use with obp, #92)

• Fix some bugs and Refactoring

  • Epsilon-greedy algorithm (#107)
  • Type checks in OPE estimators (#106)
  • Linear and logistic policies (#91)

• Welcome new contributors (#94)

references

• [1] James McInerney, Brian Brost, Praveen Chandar, Rishabh Mehrotra, and Benjamin Carterette. 2020. Counterfactual Evaluation of Slate Recommendations with Sequential Reward Interactions. In Proceedings of the 26th ACM SIGKDD
  International Conference on Knowledge Discovery and Data Mining. 1779–1788.
• [2] Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. 2018. More robust doubly robust off-policy evaluation. In Proceedings of the 35th International Conference on Machine Learning, PMLR 80, 1447–1456.

The changes are summarized below:

• Implemented obp/tests and Continuous Integration (Github Actions). Now you are free to contribute to the project! See our contribution guidelines
• Added examples describing the usage of the ReplayMethod to evaluate adaptive bandit algorithms (#67)
  • https://github.com/st-tech/zr-obp/blob/master/examples/quickstart/online.ipynb
• Made many minor fixes to improve the usage of the package as listed below
  • #77, #69, #68, #62, #59,

The changes are summarized below:

• add sample_action method to obp.policy.IPWLearner which trains an offline bandit policy that samples a non-repetitive set of actions for new data. Thus, it can be used in practice even when the action interface has a list structure
  • #22
  • detailed description: https://zr-obp.readthedocs.io/en/latest/_autosummary/obp.policy.offline.html#module-obp.policy.offline
• fix a bug in the fit_predict method of obp.ope.RegressonModel
  • #23
• Complete the benchmark experiments on a wide variety of OPE estimators using the full size of Open Bandit Dataset.
  • The detailed results and discussions can be found at the coming arXiv updates.
  • https://github.com/st-tech/zr-obp/tree/master/benchmark/ope

This release enhances the OBP package in the following ways.

• add some new contents to the obp document: https://zr-obp.readthedocs.io/en/latest/index.html
  • In particular, you can use "off-policy evaluation" section as a textbook about this area
• add obp.dataset.MultiClassToBanditReduction class for handling multi-class classification datasets as bandit feedback #19
  • https://zr-obp.readthedocs.io/en/latest/_autosummary/obp.dataset.multiclass.html#module-obp.dataset.multiclass
  • this will allow researchers to run their synthetic experiments with some multi-class classification datasets easily
  • relevant quickstart and example will be added to the repository soon
• add continuous reward option to obp.dataset.SyntheticBanditDataset
• add squared error (se) option for the evaluation of OPE with obp.ope.OffPolicyEvaluation
• fix some README and docstring inconsistencies
• refactor the dataset and ope modules

In this release, we fix some bugs in the cross-fitting procedure

This release enhances the OBP package in the following ways.

• allowing evaluation policy to be stochastic, which makes the package more consistent with the formulation of OPE
• adding some advanced estimation techniques such as cross-fitting and doubly robust with shrinkage
• modifying examples to evaluate offline bandit policies (not online ones), which again makes the package more consistent with the formulation of OPE: https://github.com/st-tech/zr-obp/tree/master/examples
• adding some slides: https://github.com/st-tech/zr-obp/tree/master/slides