580 Frederick Avenue is significant for its role in the development of the medical profession within Detroit’s Black community in the early twentieth century. The house also later became associated with two of Michigan’s first Black elected officials to hold high political office. Built in 1892 by the Guy W. Vinton Construction Company, the residence was originally constructed for jeweler and real estate developer Charles W. Warren.

At the time, Frederick Avenue was part of a fashionable residential district east of Woodward Avenue, near the homes of prominent Detroit figures such as David Whitney, Charles Lang Freer, and Frank J. Hecker. As Detroit’s population grew and racial segregation limited access to medical care, Black physicians formed the Allied Medical Society to establish a hospital serving the city’s Black residents.

In 1916 they purchased the Warren house and opened Dunbar Memorial Hospital the following year, Detroit’s first hospital serving the Black community. Dr. Ossian Sweet practiced medicine at Dunbar Hospital after earning his medical degree at Howard University. The hospital expanded into the neighboring house in 1924 before relocating to a larger facility in 1928, later known as Parkside Hospital. Soon afterward, the house was purchased by Charles C. Diggs Sr., who became Michigan’s first Black Democratic state senator in 1937. His son, Charles C. Diggs Jr., later became Michigan’s first Black member of the U.S. House of Representatives in 1954.

In the 1970s, Hilanius Phillips, who was the city's first black head city planner, championed the establishment of the Frederick Avenue Historic District, which included structures of great importance to the African American community in Detroit and to the development of the Detroit Cultural Center. The building was designated a Michigan State Historic Site in 1977. Since then, multiple attempts to convert the home into a museum have stalled.