Psalm 93 – Skeptics Annoted Bible answered

According to the reference of the author of the SAB, this quote from Calvin is taken from his commentary on Genesis. However, it is not found there. It's just made up (see also The Foundations of Dialogue in Science and Religion (page 16) by Alister E. McGrath). With a source like that, there's little need to spend more time on Andrew Dickson White.
But it is clear that Calvin believed in a geocentric universe from his commentary on this particular verse:

How could the earth hang suspended in the air were it not upheld by God’s hand? By what means could it maintain itself unmoved, while the heavens above are in constant rapid motion, did not its Divine Maker fix and establish it?

It is however unlikely he had given a heliocentric universe much thought, or had actually read Copernicus' book.
But the issue here isn't what Calvin thought, but what the text actually says. It is not probable to read this exclamation as a scientific description. The intend of the psalmist is to compare land and sea with God's might. In verse one (verse 1) the psalmist mentions the earth: it doesn't move, i.e. the ground doesn't shake, it is firm. This is contrasted in verse two (verse 2) with God's throne: that is everlasting.
That in particular the ground, the shore, the dry land is meant in verse 1 (verse 1) can be seen by the second contrast the psalmist makes in verse three (verse 3). Unlike land, the sea does move. Waves can be high and in great turmoil, appear unstoppable. But God is mightier (verse 4).
The psalmist concludes with God's testimonies: the scriptures are sure. We can trust them. The whole psalm is very similar to Ps. 19 where also nature is described in the first part, and God's kingdom in the second.
Note that John Gill takes this entire psalm as a description of the future kingdom of God, of the world to come.