Since spring 2020, the human world seems to be exceptionally silent due to mobility reduction caused by the COVID-19 pandemic. To better measure the real-time decline of human mobility and changes in socio-economic activities in a timely manner, we constructed a silent index (SI) based on Google’s mobility data. We systematically investigated the relations between SI, new COVID-19 cases, government policy, and the level of economic development. Results showed a drastic impact of the COVID-19 pandemic on increasing SI. The impact of COVID-19 on human mobility varied significantly by country and place. Bi-directional dynamic relationships between SI and the new COVID-19 cases were detected, with a lagging period of one to two weeks. The travel restriction and social policies could immediately affect SI in one week; however, could not effectively sustain in the long run. SI may reflect the disturbing impact of disasters or catastrophic events on the activities related to the global or national economy. Underdeveloped countries are more affected by the COVID-19 pandemic.