The Palace Museum (formerly the People's Palace, and before that the Sultan's Palace) was constructed on part of the site of an even older palace called Beit el Sahel, the House of the Coast, which was originally built for Sultan Said between 1827 and 1834. Contemporary accounts describe Beit el Sahel as a two-storey whitewashed palace, with a roof of green and red tiles, separated from the beach by a high wall, with a grove of pomegranates behind. The accounts go on to describe how Sultan Said spent three days of each week at Beit el Sahel, and the rest of the time at his country palace at Mtoni, about 5km north of Zanzibar Town. He often walked from the town to Mtoni even though his stables were full of Arabian horses. Every morning, the best horses were brought out from the stables and fastened to the seaward side of the wall with long ropes, to roam about and wade in the soft sand at low tide.

Another palace, called Beit el Hukm (the House of Government), was built later behind Beit el Sahel. Then, in 1883, Beit el Ajaib (the House of Wonders) was also built. These three adjoining palaces were connected by a series of covered ways and passages. A lighthouse in front of the palaces was nicknamed the 'Sultan's Christmas tree' by British navy officers, on account of its many rows of lamps. Beit el Sahel, Beit el Hukm and the lighthouse were all destroyed in the bombardment of 1896. The palace that exists today (now the museum) was constructed partly on the site of Beit el Sahel. On the site of Beit el Hukm a private house was built, which is now the offices of Stone Town Conservation and Development Authority, easily seen between the Palace Museum and the House of Wonders, set back from the road. The building has a well-maintained garden with palm trees and shrubs. Outside the main entrance is a pair of cannons, made in Boston, Massachusetts, in 1868.