The Fornax Deep Survey (FDS) with VST XII: Low surface brightness dwarf galaxies in the Fornax cluster

In this work we use Max-Tree Objects, (MTO) on the FDS data in order to detect previously undetected Low surface brightness (LSB) galaxies. After extending the existing Fornax dwarf galaxy catalogs with this sample, our goal is to understand the evolution of LSB dwarfs in the cluster. We also study the contribution of the newly detected galaxies to the faint end of the luminosity function. We test the detection completeness and parameter extraction accuracy of MTO. We then apply MTO to the FDS images to identify LSB candidates. The identified objects are fitted with 2D S\'ersic models using GALFIT and classified based on their morphological appearance, colors, and structure. With MTO, we are able to increase the completeness of our earlier FDS dwarf catalog (FDSDC) 0.5-1 mag deeper in terms of total magnitude and surface brightness. Due to the increased accuracy in measuring sizes of the detected objects, we also add many small galaxies to the catalog that were previously excluded as their outer parts had been missed in detection. We detect 265 new LSB dwarf galaxies in the Fornax cluster, which increases the total number of known dwarfs in Fornax to 821. Using the extended catalog, we show that the luminosity function has a faint-end slope of -1.38+/-0.02. We compare the obtained luminosity function with different environments studied earlier using deep data but do not find any significant differences. On the other hand, the Fornax-like simulated clusters in the IllustrisTNG cosmological simulation have shallower slopes than found in the observational data. We also find several trends in the galaxy colors, structure, and morphology that support the idea that the number of LSB galaxies is higher in the cluster center due to tidal forces and the age dimming of the stellar populations. The same result also holds for the subgroup of large LSB galaxies, so-called ultra-diffuse galaxies.