Frazier Mountain was the first lookout site for what was the Santa Barbara National Forest. A lookout was in place some time before 1920. California Region 5 Plan 4A or 4AR was constructed on the mountain. The current structure was constructed in 1943, a California Region 5 Plan BC-301 on an 11 foot timber tower. The tower is somewhat unique in that the lower enclosed room is built inside the tower structure leaving it exposed to the elements.

A Forest Service Bulletin reported that in 1932, the lookout operator received the newspaper twice daily by air. A Varney Airlines (predecessor to United Airlines) pilot dropped a San Francisco paper on his southbound route and a Los Angeles paper on his return route north. The pilot and lookout corresponded regularly via mail.