The college was founded in 1921, opening its doors to students in September of that year. SJCC is one of the oldest colleges in the California Community College System.

In 1953, San José Unified School District took over the college's operation in 1953 from San José State University. The college moved to its present location in the Fruitdale neighborhood of West San Jose in the same year.

The college's name changed to “San José City College" in 1958.

In 1999, 2004 and 2010 voters within the San José-Evergreen Community College District passed bond measures to re-build the campus and provide modern technology and facilities for the students, which resulted in the construction of buildings like César E. Chávez Library, the Science Complex, Carmen Castellano Fine Arts Center, and the SJCC Student Center.

SJCC's campus is located in West San Jose, in the neighborhood of Fruitdale. It is bound by Bascom Ave to the west, Leigh Ave to the east, and Moorpark Ave to the north.

Notable buildings on campus include César E. Chávez Library, the Science Complex, the Student Center, Carmen Castellano Fine Arts Center, and the Technology Center, among others.