Help
RSS
API
Feed
Maltego
Contact
Domain > lethalletham.com
×
More information on this domain is in
AlienVault OTX
Is this malicious?
Yes
No
DNS Resolutions
Date
IP Address
2021-01-08
52.217.78.43
(
ClassC
)
2021-01-22
52.217.71.195
(
ClassC
)
2025-08-31
16.15.195.243
(
ClassC
)
2025-09-28
16.15.187.185
(
ClassC
)
2026-01-04
16.15.183.168
(
ClassC
)
2026-01-29
52.216.218.189
(
ClassC
)
Port 80
HTTP/1.1 200 OKx-amz-id-2: D6EwxVw1bPTxLGBMCqO7IOSRS44AJCpMyj5gcZcAqTGKNuMzaQBjrn/gtbqKom+Vc0Pt5RqIeyuoT5PRq8/o33k2JvAXaJ5Kx-amz-request-id: 8390ZP5XND0N5P98Date: Thu, 29 Jan 2026 22:01:19 GMTLast-Modified: Tue, 22 Aug 2023 17:59:29 GMTETag: ed3d55ffee0a79d433d15883d00975a3Content-Type: text/htmlContent-Length: 43112Server: AmazonS3 !DOCTYPE HTML PUBLIC -//W3C//DTD HTML 4.01 Transitional//EN http://www.w3.org/TR/html4/loose.dtd>html>head>title>Ben Letham - Meta data science/title>meta namedescription contentBen Letham - Meta data science>meta propertyog:title contentBen Letham - Meta data science />meta propertyog:type contentwebsite />meta propertyog:image contenthttp://lethalletham.com/bl_pic.jpg />meta propertyog:image:width content139 />meta propertyog:image:height content159 />meta propertyog:url contenthttp://lethalletham.com/ />meta propertyfb:app_id content319087881769395 />script> (function(i,s,o,g,r,a,m){iGoogleAnalyticsObjectr;irir||function(){ (ir.qir.q||).push(arguments)},ir.l1*new Date();as.createElement(o), ms.getElementsByTagName(o)0;a.async1;a.srcg;m.parentNode.insertBefore(a,m) })(window,document,script,https://www.google-analytics.com/analytics.js,ga); ga(create, UA-831516-5, auto); ga(send, pageview);/script>script srcjquery-1.4.2.min.js>/script>script srcjquery.cross-slide.min.js>/script>!--first we define some css styles-->style typetext/css>#slideshow { width: 530px; height: 180px;}/style>style typetext/css mediascreen>ul.tabNavigation { font-family: sans-serif; font-weight: bold; margin: 0; padding: 0; }ul.tabNavigation li {display: inline;}ul.tabNavigation li a { padding: 2px 2px; background-color: #E0E0E0; border: 4px #E0E0E0 solid; color: #000; text-decoration: none; }ul.tabNavigation li a:hover { border:4px black solid; -webkit-transition-duration: 1s; /*Webkit: Animation duration*/ -moz-transition-duration: 1s; /*Mozilla duration version*/ -o-transition-duration: 1s; /*Opera duration version*/ }ul.tabNavigation li a.selected { background-color: #333; border:4px black solid; color: #fff; }ul.tabNavigation li a:focus { outline: 0; }div.tabs > div { padding: 10px; margin-top: 7px; border: 5px solid #333; font-family: sans-serif; font-size: 16px; }h1 {padding: 0px;margin-top: 0px;margin-bottom: 0px;font-size: 18px;font-weight: bold;font-family: sans-serif;color: black ;}h2 {margin-left: 40px;margin-top: 20px;margin-bottom: 0px;font-style: italic;font-family: sans-serif;font-size: 16px;padding: 0px;font-weight: bold;color: black;}h3 {margin-left: 40px;margin-right: 40px;margin-top: 0px;margin-bottom: 0px;font-weight: normal;font-style: normal;font-family: sans-serif;font-size: 16px;padding: 0px;}h4 {margin-left: 20px;margin-top: 0px;margin-bottom: 10px;font-weight: bold;font-family: sans-serif;font-size: 16px;padding: 0px;}h5 {margin-left: 20px;width: 600px;margin-top: 0px;margin-bottom: 0px;font-weight: normal;font-style: normal;font-family: sans-serif;font-size: 16px;padding: 0px;}.dpost {margin-left:20px;border-spacing:20px;}div.papers {font-family: serif;font-weight: normal;font-size: 14px;background-color: #eee;padding: 6px;margin-top: 10px;margin-right: 40px;margin-left: 40px;margin-bottom: 20px;text-align: left;border: 2px gray solid;}div.news {font-weight: normal;font-size: 14px;background-color: #eee;padding: 6px;margin-top: 20px;margin-right: 25px;margin-left: 25px;margin-bottom: 20px;text-align: left;border: 2px gray solid;}/style>script typetext/javascript>$(function () {var tabContainers $(div.tabs > div);tabContainers.hide().filter(:first).show();$(div.tabs ul.tabNavigation a).click(function () {tabContainers.hide();tabContainers.filter(this.hash).show();$(div.tabs ul.tabNavigation a).removeClass(selected);$(this).addClass(selected);return false;}).filter(:first).click();});/script>/head>body>script> window.fbAsyncInit function() { FB.init({ appId : 319087881769395, xfbml : true, version : v2.7 }); }; (function(d, s, id){ var js, fjs d.getElementsByTagName(s)0; if (d.getElementById(id)) {return;} js d.createElement(s); js.id id; js.src //connect.facebook.net/en_US/sdk.js; fjs.parentNode.insertBefore(js, fjs); }(document, script, facebook-jssdk));/script>!-- and now on to the actual content (if it can be called such...)-->!-- Well have the header in a table even though its bad form...-->table cellpadding20>tr>td>div idslideshow>/div>!--and now build a slideshow inside that div tag-->script> $(function() { $(#slideshow).crossSlide({ sleep: 8, fade: 2 }, { src: sf1.jpg }, { src: sf2.jpg }, { src: sf7.jpg }, { src: sf6.jpg }, { src: sf11.jpg }, { src: sf8.jpg }, { src: sf9.jpg }, { src: sf12.jpg }, { src: sf5.jpg }, { src: sf4.jpg }, { src: sf10.jpg }, { src: sf3.jpg }, { src: sf13.jpg }, { src: sf14.jpg }, { src: sf15.jpg } ) });/script>/td>td stylevalign:center>span stylefont-family: sans-serif; font-size: 24px>Ben Lethambr>/span>span stylefont-family: sans-serif; font-size: 18px>Central Applied Science, Meta./span>/td>/tr>/table>!-- Now the content of this page-->br>br>!--well use another table here-->div classtabs> ul classtabNavigation> li>a href#home>Home/a>/li> li>a href#research>Research/a>/li> li>a href#posts>Posts/a>/li> li>a href#other>Etc./a>/li> /ul>div idhome>table cellpadding20>tr>td> !-- here goes my picture.-->img altBen Letham srcbsqsm.jpg>/td>td>I am in the a hrefhttps://research.facebook.com/teams/central-applied-science/>Central Applied Science/a> group at Meta (formerly known as Facebook). I build statistics and optimization tools, develop new ways of analyzing data, and run large online experiments. Prior to this I finished my PhD in operations research at MIT. I can be contacted at bletham@meta.com./td>/tr>tr>td>/td>td>i>b>News from the last year:/b>/i>br>br>i>June 2023/i>: When asking humans about preferences or perception, you can also measure response time. We have submitted a paper about how to incorporate this information into a nonparametric response surface model, now on arxiv! (a hrefhttps://arxiv.org/abs/2306.06296 target_blank>arxiv/a>)br>br>i>April 2023/i>: We wrote a paper on using regularization to find sparse solutions in Bayesian optimization, now published at AISTATS! (a hrefhttps://proceedings.mlr.press/v206/liu23b/liu23b.pdf target_blank>paper/a>)br>br>i>March 2023/i>: Nonparametric response surface modeling was part of a platform for understanding hardware requirements for distortion-free rendering in AR and VR. The rest of the platform is very impressive. (a hrefhttps://arxiv.org/abs/2303.15666 target_blank>arxiv/a>)br>br>i>February 2023/i>: I made the list of top reviewers for AISTATS a hrefhttp://aistats.org/aistats2023/reviewers.html>here/a>. I guess I should ask for a raise next year!br>br>i>February 2023/i>: We have a paper published at AAAI on combining Gaussian process models with parametric models from psychology, a great way to combine domain knowledge with flexible nonparametric modeling. (a hrefhttps://ojs.aaai.org/index.php/AAAI/article/download/25074/24846 target_blank>paper/a>)br>br>i>December 2022/i>: We had a paper about personalized experimentation at the NeurIPS Workshop on Causal Inference Challenges in Sequential Decision Making. (a hrefhttps://arxiv.org/abs/2303.17648 target_blank>arxiv/a>)br>br>!-- - i>July 2022/i>: I presented our work on multi-task optimization in ranking systems at the first a hrefhttps://automl.cc/ target_blank>AutoML Conference/a> in Baltimore. It was a great conference!br>br> -->!-- - i>March 2022/i>: The science version of the trailer reaction videos that plague youtube: Paper reaction videos! Experience the excitement of watching someone else watch something, in this case, our level-set estimation paper! (a hrefhttps://www.youtube.com/watch?vdx0TBmM4wcI&t1s target_blank>youtube/a>)br>br> -->!-- - i>March 2022/i>: We wrote a paper on using regularization to find sparse solutions in Bayesian optimization, submitted but now on arxiv! (a hrefhttps://arxiv.org/pdf/2203.01900.pdf target_blank>arxiv/a>)br>br> -->!-- - i>January 2022/i>: We did some work on level-set estimation with Bernoulli response data that was published at AISTATS (a hrefhttps://proceedings.mlr.press/v151/letham22a/letham22a.pdf target_blank>pdf/a>).br>br> -->!-- - i>January 2022/i>: A significant update to our paper on transfer learning in Bayesian optimization is now on arxiv (a hrefhttps://arxiv.org/pdf/1802.02219.pdf target_blank>arxiv/a>).br>br> -->!-- - i>July 2021/i>: I reviewed papers for both ICML and NeurIPS this year.br>br> -->!-- - i>April 2021/i>: Nonparametric surrogate modeling and active learning can be really helpful in psychophysics. Weve released an open source toolbox AEPsych for adaptive psychophysics, along with a paper preprint (a hrefhttps://github.com/facebookresearch/aepsych target_blank>github/a>, a hrefhttps://arxiv.org/abs/2104.09549 target_blank>arxiv/a>).br>br> -->!-- - i>November 2020/i>: Do you suffer from high-dimensional Bayesian optimization? Alebo can help! I created an if AI papers were pharmaceuticals parody commercial (a hrefhttps://youtu.be/fbEwVxXYSok target_blank>YouTube/a>).br>br> -->!-- - i>October 2020/i>: Im thrilled that three papers Ive worked on were accepted at Neural Information Processing Systems: on linear embeddings for high-dimensional Bayesian optimization (a hrefhttps://papers.nips.cc/paper/2020/hash/10fb6cfa4c990d2bad5ddef4f70e8ba2-Abstract.html target_blank>paper/a>), on contextual optimization using only aggregate policy rewards (a hrefhttps://papers.nips.cc/paper/2020/hash/faff959d885ec0ecf70741a846c34d1d-Abstract.html target_blank>paper/a>), and on the BoTorch framework for Bayesian optimization in PyTorch (a hrefhttps://papers.nips.cc/paper/2020/hash/f5b1b89d98b7286673128a5fb112cb9a-Abstract.html target_blank>paper/a>).br>br> -->!-- - i>September 2020/i>: Another Presidential election is coming up, and that means Debate Bingo! I extracted common phrases from recent Biden and Trump speeches to make this years a hrefposts/debatebingo2020.html>bingo cards/a>.br>br> -->!-- - i>July 2020/i>: Reviewing for NeurIPS! Already excited for the conference.br>br> -->!-- - i>June 2020/i>: Im on the program committee for the a hrefhttps://sites.google.com/view/automl2020/home>AutoML workshop/a> at ICML. Its going to be a great program!br>br> -->!-- - i>February 2020/i> Im on the program commitee for the a hrefhttps://sites.google.com/view/nas2020/>Neural Architecture Search (NAS) workshop/a> at ICLR. Reviewed some really interesting papers!br>br> -->!-- - i>January 2020/i>: We did some work to understand the behavior and improve the performance of linear embeddings for high-dimensional Bayesian optimization. The first version was rejected at ICLR, but a much improved version is now on arxiv! (a hrefhttps://arxiv.org/abs/2001.11659 target_blank>arxiv/a>) (a hrefhttps://github.com/facebookresearch/alebo target_blank>code/a>).br>br> -->!-- - i>October 2019/i>: We wrote a paper that describes the concepts and framework behind BoTorch, our software for Bayesian optimization in PyTorch (a hrefhttps://arxiv.org/abs/1910.06403 target_blank>arxiv/a>).br>br> -->!-- - i>September 2019/i>: Our paper on combining online experiments with an offline simulator to tune News Feed ranking was published in JMLR (a hrefhttp://jmlr.org/papers/volume20/18-225/18-225.pdf target_blank>pdf/a>).br>br> -->!-- - i>August 2019/i>: I gave a talk in the Bayesian Optimization session at JSM.br>br> -->!-- - i>May 2019/i>: I wrote a post a hrefposts/fahrenheit.html target_blank>In Defense of Fahrenheit/a> in which I analyze data from nearly 4000 weather stations to conclude that the Fahrenheit scale is just about perfect for capturing U.S. daily temperatures.br>br> -->!-- - i>April 2019/i>: The Bayesian optimization software that I work on has been open sourced in a pair of packages, a hrefhttp://ax.dev target_blank>Ax/a> and a hrefhttp://botorch.org target_blank>botorch/a>.br>br> -->!-- - i>April 2019/i>: Weve posted a new paper to arxiv that describes how we combine online A/B tests with a naive offline simulator to tune News Feed with multi-task Bayesian optimization (a hrefhttp://arxiv.org/abs/1904.01049 target_blank>arxiv/a>) (a hrefhttps://ai.facebook.com/blog/online-and-offline-tests-to-improve-news-feed-ranking target_blank>blog post/a>).br>br> -->!-- - i>December 2018/i>: We released v0.4 of Prophet on PyPI and CRAN (a hrefhttps://facebook.github.io/prophet/ target_blank>site/a>).br>br> -->!-- - i>December 2018/i>: The Bayesian optimization tools that I develop are used for improving Instagram feed ranking, as described in this IG engineering blog post (a hrefhttps://instagram-engineering.com/lessons-learned-at-instagram-stories-and-feed-machine-learning-54f3aaa09e56 target_blank>post/a>).br>br> -->!-- - i>November 2018/i>: I gave a talk on Bayesian optimization in the University of California Santa Cruz statistics department seminar series.br>br> -->!-- - i>October 2018/i>: I gave a guest lecture on A/B testing at CU Boulder.br>br> -->!-- - i>September 2018/i>: I wrote a blog post about Bayesian Optimization at Facebook for the FB research site (a hrefhttps://research.fb.com/efficient-tuning-of-online-systems-using-bayesian-optimization/ target_blank>post/a>)br>br> -->!-- - i>September 2018/i>: Ill be on a panel at the a hrefhttps://sites.google.com/view/nipsbnp2018/ target_blank>Bayesian nonparametrics workshop/a> at NIPS in December.br>br> -->!-- - i>June 2018/i>: Our paper on transfer learning for hyperparameter optimization was accepted at the ICML AutoML workshop (a hrefhttp://arxiv.org/abs/1802.02219 target_blank>arxiv/a>).br>br> -->!-- - i>June 2018/i>: I spoke at the SLDS conference for the statistical learning section of the ASA.br>br> -->!-- - i>June 2018/i>: We just released v0.3 of Prophet with some exciting new features, now on PyPI and CRAN (a hrefhttps://facebook.github.io/prophet/ target_blank>site/a>).br>br> -->!-- - i>May 2018/i>: Our paper on Bayesian optimization for A/B testing at Facebook has been accepted at Bayesian Analysis (a hrefhttps://projecteuclid.org/euclid.ba/1533866666 target_blank>paper/a>).br>br> -->/table>/div>div idresearch>h1>Bayesian optimization and experimentation/h1>h2>Bayesian optimization with online field experiments/h2>h3>At Facebook we have many backend systems with parameters that could presumably be optimized to improve system performance. However, often the only way to measure performance is through time consuming, noisy field experiments. In this paper we describe methodological advances for using Bayesian optimization to design a set of experiments that efficiently search the parameter space. The paper also describes two systems optimizations that weve done with this approach: tuning a production ranking system, and optimizing HHVM compiler flags./h3>div classpapers>Benjamin Letham, Brian Karrer, Guilherme Ottoni, Eytan Bakshy (2019) Constrained Bayesian optimization with noisy experiments. i>Bayesian Analysis/i> 14(2): 495-519. (a hrefNoisyConstrainedBO.pdf target_blank>pdf/a>) (a hrefNoisyConstrainedBO_supplement.pdf target_blank>supplement/a>) (a hrefba2019_erratum.htm target_blank>erratum/a>) (a hrefhttps://research.facebook.com/blog/2018/09/efficient-tuning-of-online-systems-using-bayesian-optimization/ target_blank>blog post/a>)/div>h3>Ranking systems can be naively simulated offline by replaying sessions to a simulator that produces the ranked list, and then applying existing event prediction models to predict online outcomes. This approach provides a high-throughput way of testing new value model policies, but the estimates suffer from off-policy bias and related issues and end up being too inaccurate to be directly useful. In this paper we show that these biased offline estimates can be combined with online A/B tests through multi-task Bayesian optimization. Including the simulator in the optimization provides significant gains, and we study the empirical generalization behavior. We used this approach to successfully improve the News Feed ranking model./h3>div classpapers>Benjamin Letham, Eytan Bakshy (2019) Bayesian optimization for policy search via online-offline experimentation. i>Journal of Machine Learning Research/i> 20(145): 1-30. (a hrefhttp://jmlr.org/papers/volume20/18-225/18-225.pdf target_blank>pdf/a>) (a hrefhttp://jmlr.org/papers/volume20/18-225/18-225_appendix.pdf>supplement/a>) (a hrefhttps://ai.facebook.com/blog/online-and-offline-tests-to-improve-news-feed-ranking target_blank>blog post/a>)/div>h3>Our software for doing Bayesian Optimization is built on PyTorch. It is open sourced as a pair of packages (a hrefhttps://ax.dev/ target_blank>Ax/a> and a hrefhttps://botorch.org/ target_blank>BoTorch/a>), and this paper describes the framework and concepts behind using differentiable programming for Bayesian optimization./h3>div classpapers>Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, Eytan Bakshy (2020) BoTorch: a framework for efficient Monte-Carlo Bayesian optimization. In: i>Advances in Neural Information Processing Systems 33/i>, NeurIPS 20. (a hrefhttps://papers.nips.cc/paper/2020/hash/f5b1b89d98b7286673128a5fb112cb9a-Abstract.html target_blank>proceedings/a>)/div>h2>High-dimensional Bayesian optimization/h2>h3>Typical approaches for Bayesian optimization perform poorly in high dimensions. One approach for doing Bayesian optimization in high dimensions is to do the optimization in a low-dimensional embedding, but this approach can perform badly even on problems that meet the underlying assumptions. We studied the failure points for linear embedding methods, and developed techniques for improving performance and reliability of Bayesian optimization in a linear embedding./h3>div classpapers>Benjamin Letham, Roberto Calandra, Akshara Rai, Eytan Bakshy (2020) Re-examining linear embeddings for high-dimensional Bayesian optimization. In: i>Advances in Neural Information Processing Systems 33/i>, NeurIPS 20. (a hrefhttps://papers.nips.cc/paper/2020/hash/10fb6cfa4c990d2bad5ddef4f70e8ba2-Abstract.html target_blank>proceedings/a>) (a hrefhttps://youtu.be/fbEwVxXYSok target_blank>video/a>)/div>h3>One common setting that produces high-dimensional optimization problems is tuning contextual policies where some number of parameters are personalized for each of several contexts. When outcomes can only be measured at some high-level aggregate (as in an A/B testing setting), the entire policy must be tuned jointly, which produces a high-dimensional problem. However, this problem has particular structure that we can take advantage of to construct new models that we show are able to find good contextual policies./h3>div classpapers>Qing Feng, Benjamin Letham, Hongzi Mao, Eytan Bakshy (2020) High-dimensional contextual policy search with unknown context rewards using Bayesian optimization. In: i>Advances in Neural Information Processing Systems 33/i>, NeurIPS 20. (a hrefhttps://papers.nips.cc/paper/2020/hash/faff959d885ec0ecf70741a846c34d1d-Abstract.html target_blank>proceedings/a>)/div>h3>In real systems, interpretability is often very important, especially in high dimensions. We show how regularization and ideas from multi-objective optimization can be used to find sparse solutions that lend themselves to interpretability./h3>div classpapers>Sulin Liu, Qing Feng, David Eriksson, Benjamin Letham, Eytan Bakshy (2023) Sparse Bayesian optimization. In: i>Proceedings of the 26th International Conference on Artificial Intelligence and Statistics/i>, AISTATS 23, pages 3754-3774. (a hrefhttps://proceedings.mlr.press/v206/liu23b/liu23b.pdf target_blank>pdf/a>) (a hrefhttps://github.com/facebookresearch/SparseBO target_blank>code/a>)/div>h2>Hyperparameter optimization/h2>h3>Hyperparameter optimization of machine learning models is a popular application of Bayesian optimization. In a machine learning platform we may have many similar models that are being optimized, and we may want to use the results of these optimizations to warm-start others. Here we developed a method for doing this, and used it for hyperparameter optimization in Facebooks computer vision platform./h3>div classpapers>Matthias Feurer, Benjamin Letham, Eytan Bakshy (2018) Scalable meta-learning for Bayesian optimization. In: i>ICML AutoML Workshop/i>. (a hrefhttps://arxiv.org/abs/1802.02219 target_blank>arxiv/a>)/div>h3>A significantly expanded and improved version of this paper is now on arxiv!/h3>div classpapers>Matthias Feurer, Benjamin Letham, Frank Hutter, Eytan Bakshy (2022) Practical transfer learning for Bayesian optimization. i>Preprint/i>. (a hrefhttps://arxiv.org/abs/1802.02219 target_blank>arxiv/a>)/div>h2>Psychophysics/h2>h3>A common problem in psychophysics is identifying perceptual thresholds, the lowest stimulus intensity at which the stimulus is perceivable, often with respect to several other context parameters (like stimulus frequency, shape, etc.). Surrogate modeling with a GP and active learning of a similar flavor to Bayesian optimization can be highly effective for exploring these high-dimensional perception response surfaces and globally identifying thresholds. Weve done work on making this easy to do, via an open-source toolbox AEPSych and an associated paper./h3>div classpapers>Lucy Owen, Jonathan Browder, Benjamin Letham, Gideon Stocek, Chase Tymms, Michael Shvartsman (2021) Adaptive nonparametric psychophysics. i>Preprint/i>. (a hrefhttps://github.com/facebookresearch/aepsych target_blank>github/a>, a hrefhttps://arxiv.org/abs/2104.09549 target_blank>arxiv/a>)/div>h3>Identifying perceptual thresholds is a level-set estimation (LSE) problem. The best acquisition functions for LSE, look-ahead acquisition, relied on update formulae that were no longer analytical when working with Bernoulli observations, which are common in psychophysics. We developed a new class of look-ahead acquisition functions for Bernoulli observations that significantly improved active learning performance./h3>div classpapers>Benjamin Letham, Phillip Guan, Chase Tymms, Eytan Bakshy, Michael Shvartsman (2022) Look-ahead acquisition functions for Bernoulli level set estimation. In: i>Proceedings of the 25th International Conference on Artificial Intelligence and Statistics/i>, AISTATS 22, pages 8493-8513. (a hrefhttps://proceedings.mlr.press/v151/letham22a/letham22a.pdf target_blank>pdf/a>) (a hrefhttps://github.com/facebookresearch/bernoulli_lse target_blank>code/a>) (a hrefhttps://www.youtube.com/watch?vdx0TBmM4wcI target_blank>video/a>)/div>h3>There is a large and old literature in psychology on measuring perceptual thresholds. We can combine parametric models from psychology with nonparametric GPs to get the best of both worlds./h3>div classpapers>Stephen Keeley, Benjamin Letham, Craig Sanders, Chase Tymms, Michael Shvartsman (2023) A semi-parametric model for decision making in high-dimensional sensory discrimination tasks. In: i>Proceedings of the 37th AAAI Conference on Artificial Intelligence/i>, AAAI 23, pages 40-47. (a hrefhttps://ojs.aaai.org/index.php/AAAI/article/download/25074/24846 target_blank>pdf/a>)/div>h3>When asking humans their preferences or perception, we can measure reaction times alongside their response. This can be helpful for improving response surface models in parts of the parameter space where the human is uncertain about their preference/perception, in which case they often take longer to respond./h3>div classpapers>Michael Shvartsman, Benjamin Letham, Stephen Keeley (2023) Response time improves choice prediction and function estimation for Gaussian process models of perception and preferences. i>Submitted/i>. (a hrefhttps://arxiv.org/abs/2306.06296 target_blank>arxiv/a>)/div>h3>One of the applications of these psychophysics models is setting hardware requirements for AR and VR. This paper describes a platform for doing that, in which nonparametric modeling played a role./h3>div classpapers>Phillip Guan, Eric Penner, Joel Hegland, Benjamin Letham, Douglas Lanman (2023) Perceptual requirements for world-locked rendering in AR and VR. i>Submitted/i>. (a hrefhttps://arxiv.org/abs/2303.15666 target_blank>arxiv/a>)/div>br>h1>Forecasting/h1>h2>The Prophet forecasting package/h2>h3>Forecasting is a common data science task, yet also a specialized skill outside the expertise of many data scientists. The Prophet forecasting package is designed to be flexible enough to handle a range of business time series, while still being configurable by non-experts. We developed it for a collection of important forecasting tasks at Facebook, and have since open sourced it. It is built on Stan and has R and Python versions. It has received a tremendous response, filling a real need for forecasting these types of time series. There is code, a site with documentation, and a paper./h3>div classpapers>Sean J. Taylor, Benjamin Letham (2018) Forecasting at scale. i>The American Statistician/i> 72(1): 37-45. (a hrefForecastingAtScale.pdf target_blank>pdf/a>) (a hrefhttps://github.com/facebook/prophet target_blank>code/a>) (a hrefhttps://facebook.github.io/prophet/ target_blank>site/a>)/div>br>h1>Nonlinear dynamics and computational biology/h1>h2> Prediction uncertainty and optimal experimental design/h2>h3>During the work below on modeling the immune response to HIV infection, we ran into two questions for which we couldnt find a satisfactory answer. The first was to determine the extent to which the collected data constrained the model predictions (prediction uncertainty). The second was to figure out what additional experiments would be most helpful for reducing that uncertainty (optimal experimental design). While eating a hamburger one weekend, I had an idea for how both of these questions might be answered by a particular optimization problem. After hashing out the details, running some experiments, and doing some theoretical work, we wrote it up in this paper./h3>div classpapers>Benjamin Letham, Portia A. Letham, Cynthia Rudin, Edward P. Browne (2016) Prediction uncertainty and optimal experimental design for learning dynamical systems. i>Chaos/i> 26: 063110. (a hrefprediction_uncertainty.pdf target_blank>pdf/a>) (a hrefpu_erratum.pdf target_blank>erratum/a>)/div>h2>The immune response to HIV infection/h2>h3>I provided computational and statistical expertise for this study of the dynamics of HIV inhibition by interferon./h3>div classpapers>Edward P. Browne, Benjamin Letham, Cynthia Rudin (2016) A computational model of inhibition of HIV-1 by interferon-alpha. i>PLoS ONE/i> 11(3): e0152316. (a hrefhttp://journals.plos.org/plosone/article/asset?id10.1371%2Fjournal.pone.0152316.PDF target_blank>pdf/a>) (a hrefhttp://journals.plos.org/plosone/article?id10.1371%2Fjournal.pone.0152316 target_blank>PLOS/a>)/div>br>h1>Learning from sales transaction data/h1>h2>Estimating arrivals and demand in the presence of stockouts/h2>h3> Suppose we wish to use sales transaction data to estimate demand for a collection of substitutable products. When an item goes out of stock, these data no longer reflect the original demand, since some customers leave with no purchase while others substitute alternative products for the one that was out of stock. We developed a Bayesian hierarchical model for simultaneously estimating the customer arrival rate and primary demand, which we use on data from a local bakery./h3>div classpapers>Benjamin Letham, Lydia M. Letham, and Cynthia Rudin (2016) Bayesian inference of arrival rate and substitution behavior from sales transaction data with stockouts. In: i>Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining/i>, KDD 16. (a hrefhttps://www.kdd.org/kdd2016/papers/files/rpp0003-lethamA.pdf target_blank>pdf/a>) (a hrefhttps://youtu.be/XmetwslSNGY target_blank>video/a>)/div>h2>Bundle pricing from retail transaction data/h2>h3>Bundle offers are often a low-risk way for retailers to increase their profits, if they are priced correctly. We show how historical sales data for the individual items in the bundle can be leveraged to predict how profits will be affected by introducing a bundle at a particular price. We use a copula model that considers the correlations in item demands, and requires far less stringent assumptions than past work in bundling. This work was done during my summer internship at IBM Research./h3>div classpapers>Benjamin Letham, Wei Sun, and Anshul Sheopuri (2014) Latent variable copula inference for bundle pricing from retail transaction data. In: i>Proceedings of the 31st International Conference on Machine Learning/i>, ICML 14, pages 217-225. (a hrefhttp://proceedings.mlr.press/v32/letham14.pdf target_blank>pdf/a>) (a hreficml2014_errata.htm target_blank>errata/a>)/div>br>h1>Predictions from rules/h1>h2>Learning a decision list to predict stroke risk/h2>h3>A decision list is a particular way of organizing association rules to form a classifier which can be both powerful and interpretable. We developed a Bayesian approach for fitting decision lists and used it to create a simple system for predicting stroke risk in atrial fibrillation patients (a la a hrefhttp://en.wikipedia.org/wiki/CHADS2_score>CHADS2/a>)./h3>div classpapers>Benjamin Letham, Cynthia Rudin, Tyler H. McCormick, and David Madigan (2015) Interpretable classifiers using rules and Bayesian analysis: building a better stroke prediction model. i>Annals of Applied Statistics/i> 9(3): 1350-1371. (a hrefhttps://projecteuclid.org/download/pdfview_1/euclid.aoas/1446488742 target_blank>pdf/a>) (a hrefBRL_supplement_code.zip target_blank>Python code/a>)/div>h2>Sequential event prediction/h2>h3>Here we discuss using techniques from supervised ranking to make predictions on data that are sequentially revealed, using a model that is a linear combination of rules. It makes predictions using only the set of past events, not their order, and is demonstrated with email recipient recommendation, medical symptom prediction, and an e-commerce recommender system./h3>div classpapers>Benjamin Letham, Cynthia Rudin, and David Madigan (2013) Sequential event prediction. i>Machine Learning/i> 93: 357-380. (a hrefhttps://link.springer.com/content/pdf/10.1007/s10994-013-5356-5.pdf>pdf/a>)/div>h2>Association rules, collaborative filtering, and recommender systems/h2>h3>The a hrefhttp://www.ecmlpkdd2013.org/discovery-challenge/>ECML PKDD Discovery Challenge 2013/a> was to develop a recommender system for recommending given names to parents looking for the perfect baby name at nameling.net (a now defunct site). For the competition, I developed a recommender system based on association rules, combined with ideas from user-based collaborative filtering. Despite its simplicity and a lack of feature engineering, I won 3rd place for the offline challenge. A major advantage of my approach was that it was fast enough that recommendations could be made entirely online, and I won 2nd place in an online challenge, where my recommender system was actually implemented into the nameling website./h3>div classpapers>Benjamin Letham (2013) Similarity-weighted association rules for a name recommender system. In: i>Proceedings of the ECML PKDD 2013 Discovery Challenge Workshop/i>. (a hrefLetham_SimConf.pdf target_blank>pdf/a>)/div>h2>Learning theory and association rules/h2>h3>We developed a learning theory framework for using association rules for classification and sequential event prediction. We use both VC bounds and results from algorithmic stability. The work was first published at COLT, and then an extended version at JMLR./h3>div classpapers>Cynthia Rudin, Benjamin Letham, Ansaf Salleb-Aouissi, Eugene Kogan and David Madigan (2011) Sequential event prediction with association rules. In: i>Proceedings of the 24th Annual Conference on Learning Theory/i>, COLT 11, pages 615-634. (a hrefrudin11a.pdf target_blank>pdf/a>)/div>div classpapers>Cynthia Rudin, Benjamin Letham, David Madigan (2013) Learning theory analysis for association rules and sequential event prediction. i>Journal of Machine Learning Research/i> 14: 3385-3436. (a hrefhttp://www.jmlr.org/papers/volume14/rudin13a/rudin13a.pdf target_blank>pdf/a>)/div>br>h1>Information retrieval/h1>h2>Growing a list using the Internet/h2>h3>There are many situations in which important information is scattered across the Internet in many incomplete lists. Given a few seed examples of what were interested in, can we find and compile all of these incomplete lists of related items to grow a complete list in an unsupervised way? We developed an algorithm for doing this and which works well enough to be useful on a wide range of list growing problems. This work was presented at ECML PKDD 2013./h3>div classpapers>Benjamin Letham, Cynthia Rudin, and Katherine Heller (2013) Growing a list. i>Data Mining and Knowledge Discovery/i> 27: 372-395. (a hrefLetham_GrowingAList_final.pdf>pdf/a>) (a hrefLetham_GrowingAList_final_supplement.pdf>supplement/a>)/div>h3>This project received some attention in the media, including a segment about the project on 89.7 WGBH Boston Public Radio. A recording of the live interview is a hrefhttp://www.wgbhnews.org/post/new-way-google target_blank>here/a>./h3>br>h1>Neuroscience/h1>h2> Cross-sensory interactions/h2>h3>While working at a hrefhttp://www.nmr.mgh.harvard.edu/>The Martinos Center/a> for Biomedical Imaging at Mass. General Hospital I was involved in a study of cross-sensory interactions (auditory stimuli activate visual cortex and vice versa) using MEG and fMRI./h3>div classpapers>Tommi Raij, Jyrki Ahveninen, Fa-Hsuan Lin, Thomas Witzel, Iiro P. Jaaskelainen, Benjamin Letham, Emily Israeli, Cherif Sahyoun, Christos Vasios, Steven Stufflebeam, Matti Hamalainen (2010) Onset timing of cross-sensory activations and multisensory interactions in auditory and visual sensory cortices. i>European Journal of Neuroscience/i>, 31(10): 1772-1782. (a hrefRaij2010-EurJNeurosci-AV.pdf>pdf/a>)/div>h3>The difficulties of measuring the timing of weak cross-sensory interactions led to this work:/h3>div classpapers>Benjamin Letham and Tommi Raij (2011) Statistically robust measurement of evoked response onset latencies. i>Journal of Neuroscience Methods/i>, 194(2): 374-379. (a hrefLetham2011-JNeurosciMeth-onset_latency.pdf>pdf/a>)/div>/div>div idposts>h1>Machine learning-themed videos/h1>table classdpost>tr>td> !-- here goes the picture.-->img titleAISTATS paper reaction | Mega Compilation!! srcvid1s.jpg>/td>td>h4>a hrefhttps://www.youtube.com/watch?vdx0TBmM4wcI target_blank>AISTATS paper reaction | Mega Compilation!!/a>/h4>h5>Paper reaction videos are the latest trend sweeping AI and ML. Efforts are underway to entirely replace the peer review process with the number of likes and subscribes on reaction videos.This video is a compilation of just a few of the reaction videos for a recent a hrefhttps://proceedings.mlr.press/v151/letham22a/letham22a.pdf target_blank>AISTATS paper/a>. A lot of excitement! Its gonna be great!/h5>/td>/tr>tr>td> !-- here goes the picture.-->img titleIf AI papers were pharmaceuticals (a parody commercial) srcvid2s.jpg>/td>td>h4>a hrefhttps://youtu.be/fbEwVxXYSok target_blank>If AI papers were pharmaceuticals (a parody commercial)/a>/h4>h5>Do you suffer from high-dimensional Bayesian optimization? a hrefhttps://papers.nips.cc/paper/2020/hash/10fb6cfa4c990d2bad5ddef4f70e8ba2-Abstract.html target_blank>Alebo/a> can help! Watch this commercial to understand how Alebo can make your life happier, your food tastier, and your friends funnier./h5>/td>/tr>tr>td> !-- here goes the picture.-->img titleKDD video srcvid3s.png>/td>td>h4>a hrefhttps://youtu.be/XmetwslSNGY target_blank>Stockouts and substitutions at Landos Bakery/a>/h4>h5>When KDD required me to make a video describing a hrefhttps://www.kdd.org/kdd2016/papers/files/rpp0003-lethamA.pdf target_blank>our paper/a> on stockouts and substitutions at a bakery, I decided it was the perfect opportunity to try making a stop motion video, something I had always wanted to do. It was my first but not my last, and it was every bit as fun as I imagined it would be. Perhaps not quite what the conference organizers had in mind, but I think it got the ideas across pretty well!/h5>/td>/tr>/table>br>br>h1>Data science posts/h1>table classdpost>tr>td> !-- here goes the picture.-->img titleDebate bingo 2020 Biden v. Trump srcdebate_bingo_2020_biden_trump.png>/td>td>h4>a hrefposts/debatebingo2020.html>Debate bingo: 2020 presidential edition/a>/h4>h5>The second edition of Debate Bingo! I once again extracted commonly used phrases from each candidate to create a bingo card generator, for use during the debates./h5>/td>/tr>tr>td> !-- here goes the picture.-->img titleFahrenheit srcfahrenheit.png>/td>td>h4>a hrefposts/fahrenheit.html>In Defense of Fahrenheit/a>/h4>h5>On a scale of 0 to 100, how warm is it? I combine Census data with daily temperatures from nearly 4000 weather stations to estimate the distribution of temperatures experienced by people in the U.S. The oft-maligned Fahrenheit scale nearly perfectly captures this distribution and is a great temperature scale for U.S. weather./h5>/td>/tr>tr>td> !-- here goes the picture.-->img titleDebate bingo 2016 Clinton v. Trump srcdebate_bingo_2016_clinton_trump.png>/td>td>h4>a hrefposts/debatebingo2016.html>Debate bingo: 2016 presidential edition/a>/h4>h5>Presidential candidates tend to repeat themselves, and our two for 2016 are no exception. I extracted some of their most commonly used phrases and used them to create a bingo card generator, for use during the debates./h5>/td>/tr>tr>td> !-- here goes the picture.-->img titleLukes ROTJ lightsaber. srclightsaber_mosaic.jpg>/td>td>h4>a hrefposts/lightsabers_and_talking.html>The best part of a lightsaber duel is the talking/a>/h4>h5>Why are the original Star Wars lightsaber duels so much better than those from the Prequels? Its not the simple fight choreography, its the talking. In this post I quantify the balance between talking and fighting in Star Wars lightsaber duels and show that there is a clear difference between the Original Trilogy and the Prequels. Unfortunately, i>The Force Awakens/i> is a typical Prequels duel./h5>/td>/tr>tr>td> !-- here goes my picture.-->img titleThe enormous piles of snow in my front yard. Eesh. srcsnowbank.png>/td>td>h4>a hrefposts/winter2015.html>Was 2015 Bostons worst winter yet?/a>/h4>h5>When I finally had some time after spending the entire month of February shoveling snow, I did an analysis of snowfall data to determine exactly how unusual of a winter it was. Although 2015 beat the all-time total snowfall record by only a couple inches, a deeper look at the data revealed that this was by far the worst winter in recorded history./h5>/td>/tr>/table>br>br>h1>Technical expositions/h1>h2>On NP-completeness/h2>h3>The theory of NP-completeness has its roots in a foundational result by Cook, who showed that Boolean satisfiability (SAT) is NP-complete and thus unlikely to admit an efficient solution. In this short paper I prove an analogous result using binary integer programming in the place of SAT. The proof is notationally cleaner and more straightforward to those for whom the language of integer programming is more natural than that of SAT./h3>div classpapers>Benjamin Letham (2011) An integer programming proof of Cooks theorem of NP-completeness. (a hrefLetham_NP_ZOIP.pdf target_blank>pdf/a>)/div>h2>On Bayesian Analysis/h2>h3>In this work I provide a somewhat rigorous yet simultaneously informal introduction to Bayesian analysis. It covers everything from the theory of posterior asymptotics to practical considerations of MCMC sampling. It assumes the reader is comfortable with basic probability and some mathematical rigor./h3>div classpapers>Benjamin Letham (2012) An overview of Bayesian analysis. (a hrefLetham_bayesian_analysis.pdf target_blank>pdf/a>)/div>/div>div idother>h1>About me/h1>When Im not working on research, I enjoy sailing, traveling, playing the violin, and programming. I am also a developer and maintainer of the a hrefhttps://facebookincubator.github.io/prophet/>Prophet/a> time series forecasting package for R and Python.br>br>My Erdös number is 4.br>br>/div>/div>/body>/html>
View on OTX
|
View on ThreatMiner
Please enable JavaScript to view the
comments powered by Disqus.
Data with thanks to
AlienVault OTX
,
VirusTotal
,
Malwr
and
others
. [
Sitemap
]