The Rare Book Department, as it was first known, was housed on the 4th floor of the newly constructed Memorial Library opened in the early 1950s. The Department was subsequently renamed the Department of Rare Books and Special Collections and then the Department of Special Collections, which moved into the newly constructed 8th and 9th floors of Memorial Library in 1990.
Dennis A. Hill, “The Rare Book Department of the University of Wisconsin-Madison: Origins and early developments, 1948-1960,” Transactions of the Wisconsin Academy of Sciences, Arts and Letters, v. 72 (1984), 40-48. Hill’s article includes a listing of major collections added to the Rare Book Department through 1960. (Issues of the Transactions of the Wisconsin Academy of Sciences, Arts and Letters beginning with 1870-1872 are available as part of the University of Wisconsin Digital Collections.)
Jim Feldman, The buildings of the University of Wisconsin
(The University Archives, 1997), especially pp. 271-274 about Memorial Library, digital version available in the UW Digital Collections.