Prior to European settlement and well into the 19th century the Quibaree Clan, part of the larger Kuringgai Tribe, frequented Lavender Bay.

In 1805 Robert Campbell purchased a large section of land on the waterfront of the North Shore, and James Milson settled on this land in 1806. During the 1850s, subdivision and sale led to increased settlement in the Lavender Bay and Milsons Point area.

The North Shore Rail Line (St Leonards to Hornsby) opened on 1 January 1890, and reached Waverton in 1893. The extension to Lavender Bay-Milsons Point was considered “An expensive and largely useless railway”. Nonetheless the Lavender Bay line eventually commenced in 1890 and opened in 1893. A large v-shaped beach cove at Lavender Bay was filled in to enable the extended railway line to be built. The line closed in 1924 in preparation for the Sydney Harbour Bridge, and the main rail line was routed over the bridge in 1932.