The Lake Herman area was a campsite for Indians traveling to Pipestone quarries just east of here. Camps have been uncovered, and projectile points, stone hammerheads, and other artifacts may still be found.

The first settlers at Lake Herman were Herman Luce and his son, William. They settled the timberland on the east side of the lake and assumed squatters' rights on June 17, 1870. The log cabin in the park was built for Herman Luce in the summer of 1871. The cabin is made of sturdy oak logs handhewed to fit snugly and chinked to keep out the cold. When constructed, it was in the midst of a stand of oak, ash, hackberry, walnut, and native fruit trees. The park is near the site of Herman, a village established in 1875. Luce's cabin briefly served as a U. S. Land Office, with Herman Luce in charge.

Madison purchased 118 acres of land for a park in 1941. In 1946, the land was given to the Department of Game, Fish and Parks. Later additions boosted the park to 176 acres.

Lake Herman has been a popular campsite for hundreds of years. Before white settlers entered the region, it was popular among Indians traveling to nearby Pipestone quarries. The lake was also a popular hunting camp.