Born on a small island near Martha's Vineyard, R. Swain Gifford and his family moved to the New Bedford, Massachusetts, area when he was two years old. The Dutch marine painter Albert Van Beest arrived in New Bedford in 1854, providing the teenage Gifford with the opportunity of study; he collaborated with Van Beest on a number of ship pictures and in 1857 accompanied the senior artist when he moved to New York. Nevertheless, Gifford's limited financial resources soon forced him to return to New Bedford. Seven years later he opened a studio in Boston, where he produced romantic marine paintings and views of storm-tossed ships derivative of Van Beest and another New Bedford influence, William Bradford.

By late 1865 Gifford had moved back to New York, and there he befriended the older painter Samuel Colman. They shared interests in watercolor painting and etching and together were instrumental in founding the American Society of Painters in Water Color in 1867 and the New York Etching Club in 1878. 

The 1870s were filled with travel for Gifford. He went to the West Coast in 1869 to execute illustrations for the poet William Cullen Bryant's book Picturesque America, visited Europe and North Africa in 1870-71 with Louis Comfort Tiffany, and made a more extensive European and North African sojourn in 1873-75 with his new wife, the artist Frances ("Fanny") Eliot. Gifford's travels abroad had a formative impact: he was exposed to the eastern subject matter that immediately became a specialty and introduced to the Barbizon landscape style that characterized his later views of the New England coast.

In 1877 Gifford began an affiliation with the Cooper Union schools that lasted until his death; he gradually rose from the position of painting instructor to become art director of all the Cooper Union schools.

In later years Gifford divided his time between New York and a summer home in Nonquitt, Massachusetts. He also traveled extensively, going to Florida, Canada, and Europe, and accompanying the railroad magnate E. H. Harriman on his scientific expedition to Alaska in 1899. Gifford's interest in the relationship of art and science is documented in the Academy's library register; between 1867 and 1895 he checked out books on Goethe's theory of colors, on the anatomy of expression, and on the writings of Leonardo da Vinci.

Although he was an important figure in the Society of American Artists, serving as its vice-president (1878-79) and on its advisory board (1900-1901), Gifford was also devoted to the National Academy. A regular participant in the annual exhibitions from 1863, he also served on the Council from 1886 to 1889 and from 1902 until the year of his death. A memorial exhibition of Gifford's work was organized at the Century Association, New York, in 1905.