Ammonium polyphosphate-diatomite composite filler (APP-diatomite composite filler) was modified with silane coupling agent KH550 to improve the flame retardancy of filled paper. Cone calorimeter was used to analyze the heat and smoke releasing rates, as well as smoke toxicity of the filled paper. The distribution of the composite filler particles in paper and the morphology of the charred residues after combustion were investigated by scanning electron microscope (SEM), and the chemical structure of the charred residues was studied with fourier transform infrared spectroscopy (FTIR). Results show that the peak heat releasing rate (PHRR), total heat release (THR) and peak mass loss rate (PMLR) of the filled paper with the modified APP-diatomite decreased markedly, compared with those for the control paper, while the charred residue after combustion increased. In addition, the filled paper had an increased peak rate of smoke release (RSR) and increased total smoke release (TSR) and peak CO production rates, but a decreased peak CO₂ production rate. It was also found that part of the carbon element in the charred residue of the paper loaded with the modified APP-diatomite was in the forms of C=C=C, C≡C and C≡N, and the charred residue had a relatively more intact structure without apparent fiber breakage and longitudinal cracks.