Search for nonresonant Higgs boson pair production in final states with two bottom quarks and two photons in proton-proton collisions at $\sqrt{s}=$ 13 TeV

A search for nonresonant production of Higgs boson pairs via gluon-gluon and vector boson fusion processes in final states with two bottom quarks and two photons is presented. The search uses data from proton-proton collisions at a center-of-mass energy of $\sqrt{s}=$ 13 TeV recorded with the CMS detector at the LHC, corresponding to an integrated luminosity of 137 fb$^{-1}$. No significant deviation from the background-only hypothesis is observed. An upper limit at 95% confidence level is set on the product of the Higgs boson pair production cross section and branching fraction into $\gamma\gamma\mathrm{b\overline{b}}$. The observed (expected) upper limit is determined to be 0.67 (0.45) fb, which corresponds to 7.7 (5.2) times the standard model prediction. This search has the highest sensitivity to Higgs boson pair production to date. Assuming all other Higgs boson couplings are equal to their values in the standard model, the observed coupling modifiers of the trilinear Higgs boson self-coupling $\kappa_\lambda$ and the coupling between a pair of Higgs bosons and a pair of vector bosons $c_{2\mathrm{V}}$ are constrained within the ranges $-$3.3 $\lt\kappa_\lambda\lt$ 8.5 and $-$1.3 $\lt c_{2\mathrm{V}}\lt$ 3.5 at 95% confidence level. Constraints on $\kappa_\lambda$ are also set by combining this analysis with a search for single Higgs bosons decaying to two photons, produced in association with top quark-antiquark pairs, and by performing a simultaneous fit of $\kappa_\lambda$ and the top quark Yukawa coupling modifier $\kappa_t$.