Search for Resonant and Nonresonant Higgs Boson Pair Production in the $b\overline{b}{\tau }^{+}{\tau }^{-}$ Decay Channel in $pp$ Collisions at $\sqrt{s}=13\text{}\text{}\mathrm{TeV}$ with the ATLAS Detector

A search for resonant and nonresonant pair production of Higgs bosons in the $b\overline{b}{\tau }^{+}{\tau }^{-}$ final state is presented. The search uses $36.1\text{}\text{}{\mathrm{fb}}^{-1}$ of $pp$ collision data with $\sqrt{s}=13\text{}\text{}\mathrm{TeV}$ recorded by the ATLAS experiment at the LHC in 2015 and 2016. Decays of the $\tau $-lepton pairs with at least one $\tau $ lepton decaying to final states with hadrons and a neutrino are considered. No significant excess above the expected background is observed in the data. The cross-section times branching ratio for nonresonant Higgs boson pair production is constrained to be less than 30.9 fb, 12.7 times the standard model expectation, at 95% confidence level. The data are also analyzed to probe resonant Higgs boson pair production, constraining a model with an extended Higgs sector based on two doublets and a Randall-Sundrum bulk graviton model. Upper limits are placed on the resonant Higgs boson pair production cross-section times branching ratio, excluding resonances $X$ in the mass range $305\text{}\mathrm{GeV}<{m}_{X}<402\text{}\mathrm{GeV}$ in the simplified hMSSM minimal supersymmetric model for $\mathrm{tan}\beta =2$ and excluding bulk Randall-Sundrum gravitons ${G}_{\mathrm{KK}}$ in the mass range $325\text{}\mathrm{GeV}<{m}_{{G}_{\mathrm{KK}}}<885\text{}\mathrm{GeV}$ for $k/{\overline{M}}_{\mathrm{Pl}}=1$.