Patterson was born into a family of the sea. His father was president of the Whitehaven Shipbuilding Company of Southampton, his uncle was a ship master and his brother was also a mariner. At thirteen, Patterson was apprenticed and went to sea as ship's boy.
As a deep water sailor, Patterson rounded Cape Horn four times. While sailing before the mast, he discovered his ability to depict his love of ships and the sea through painting, initially on his shipmates' sea chests. He studied art briefly in Chicago and then moved to settle and open a studio in New York while still in his twenties. His paintings of South Street are still considered the most accurate record of that seaport.
Author Richard McKay accurately wrote of Patterson that "only those who brave the dangers of the sea and know its mysteries realize the beauty and grandeur of the sea and make us see the ships of old in all their glory."