Materials inform many of our interactions with everyday objects. Knowing that a cup is ceramic, we handle it more gently. When sidewalks are covered with snow and ice, we walk differently so as not to slip. If we aim to create an autonomous system, such as a robot, that can manipulate a wide variety of objects or traverse the many different surfaces it may encounter, we will need to be able to provide this material information algorithmically. Visual material recognition is the process of identifying the presence of materials, such as plastic, glass, or metal, in ordinary images. By recognizing these materials, we can obtain valuable cues for general image understanding. Doing so, however, is a challenging problem, as a single material may exhibit many different visual appearances. We can recognize an object based on its characteristic shape, but materials do not have such a singular distinguishing property. In this thesis, we study the problem of visual material recognition by breaking the recognition process down into fundamental and separable components. Our key observation is that the appearance variation which makes materials so challenging to recognize arises from the context in which the materials appear. A smooth white surface does not on its own provide many cues as to the material in question, but when combined with the fact that the surface is on a mug, we may infer that the material is likely ceramic or plastic. In order to take advantage of this observation, we must be able to separate material appearance from the context in which it appears. As a first step, we demonstrate that it is possible to recognize materials from small image patches. These small patches contain only the appearance of the material, and not that of the surrounding context. We achieve this by using the simple visual material properties which humans use to describe materials, such as "shiny" or "translucent", as an intermediate representation for the materials themselves. We refer to these properties as visual material traits. Though they prove useful, obtaining annotations for these traits is a challenging and time-consuming process. To address this, we derive an automatic perceptual attribute discovery method that generates classifiers for a set of unknown attributes. By probing the human perception of materials through easily-obtained binary annotations, we may measure the visual similarity of materials and discover attributes that serve the same function as material traits. Finally, having shown that material appearance may be isolated in small local image patches, we introduce a convolutional neural network (CNN)-based framework that integrates local material appearance with global contextual cues. By cleanly separating and combining the material appearance and context, we can take advantage of the strong material cues we show are present in that context to accurately recognize materials with far fewer examples than past attempts at material recognition.