Context. Multiwavelength observations are indispensable in studying disk geometry and dust evolution processes in protoplanetary disks. Aims. We aim to construct a three-dimensional model of HD 163296 that is capable of reproducing simultaneously new observations of the disk surface in scattered light with the SPHERE instrument and thermal emission continuum observations of the disk midplane with ALMA. We want to determine why the spectral energy distribution of HD 163296 is intermediary between the otherwise well-separated group I and group II Herbig stars. Methods. The disk was modeled using the Monte Carlo radiative transfer code MCMax3D. The radial dust surface density profile was modeled after the ALMA observations, while the polarized scattered light observations were used to constrain the inclination of the inner disk component and turbulence and grain growth in the outer disk. Results. While three rings are observed in the disk midplane in millimeter thermal emission at ~80, 124, and 200 AU, only the innermost of these is observed in polarized scattered light, indicating a lack of small dust grains on the surface of the outer disk. We provide two models that are capable of explaining this difference. The first model uses increased settling in the outer disk as a mechanism to bring the small dust grains on the surface of the disk closer to the midplane and into the shadow cast by the first ring. The second model uses depletion of the smallest dust grains in the outer disk as a mechanism for decreasing the optical depth at optical and near-infrared wavelengths. In the region outside the fragmentation-dominated regime, such depletion is expected from state-of-the-art dust evolution models. We studied the effect of creating an artificial inner cavity in our models, and conclude that HD 163296 might be a precursor to typical group I sources.