Search for resonances decaying to an anomalous jet and a Higgs boson in proton-proton collisions at √s=13TeV

This paper presents a search for new physics through the process where a massive particle, X, decays into aHiggs boson and a second particle, Y. The Higgs boson sub-sequently decays into a bottom quark-antiquark pair, which is reconstructed as a single large-radius jet. The decay products ofY are also assumed to produce a single large-radius jet. The identification of theY particle is enhanced by computing the anomaly score of its candidate jet using an autoencoder, which measures deviations from typical quark- or gluoninduced jets. This allows a simultaneous search for multiple Y decay scenarioswithin a single analysis. In the main benchmark process, Y is a scalar particle that decays into a W boson pair. Two other scalar Y decay processes are also considered as benchmarks: decays to a light quark-antiquark pair, and decays to a top quark-antiquark pair. A fourth benchmark process considers Y as a hadronically decaying top quark, arising from the decay of a vector-like quark into a top quark and aHiggs boson. Data recorded by theCMSexperiment at a center-of-mass energy of 13 TeV in 2016-2018, corresponding to an integrated luminosity of 138 fb(-1), are analyzed. The search covers X masses between 1.4 and 3.0 TeV and Y masses between 90 and 400 GeV, with all simulated signals produced in the narrow-width approximation. No significant excess above the standard model background expectation is observed. The most stringent upper limits to date are placed on benchmark signal cross sections for various masses of X and Y particles