The minhag of tashlich dates back to at least the 14th century. The Maharil (Minhagei Rosh Hashanah 9) writes about going to rivers that contain fish in order to throw our sins into the depths of the sea (Micha 7:19). The Rema (OC 583:2) records this minhag.
The basic text is the final 3 Pesukim of Micha (7:18-20) which parallel the Thirteen Midos, Divine Attributes of Hashem. Many add more tefillos, composed primarily by the Chida.
While not all observe this minhag, R’ Ovadia Yosef (Yechave Daas 1:54) writes that this is the minhag of Sefardim too, following the Arizal.
Different reasons have been offered for this minhag:
The Maharil refers to the famous Midrash (Yalkut Shimoni, Bereishis 99). The satan tried various tactics to dissuade Avraham from carrying out the akedah including creating a river. This explains why it took them 3 days to reach Har Hamoriah. The water of tashlich serves to remind us of Avraham’s self-sacrifice (See Gra in his notes to the Rema OC 583:2).
In keeping with the various symbolic foods eaten on Rosh Hashana eve, the Rema (Darkei Moshe OC 583) writes that we should proliferate like fish that are immune to ayin hora (See Mishna Berura 583:8). Elsewhere (Torah Haolah 3:56), the Rema suggests another reason. One going to the river bank or shore observes Hashem’s majestic creation and is reminded of the world completed by Hashem on Rosh Hashana. Doing so will cause them to do teshuva and Hashem will forgive them, throwing their sins into the depths of the sea.
The Levush (OC 596) writes that tashlich should make us think of fish caught in a net as on Rosh Hashana our lives hang in the balance. This image should stir us to do teshuva.
The Elyah Rabbah (OC 596) suggests that as fish never close their eyes, tashlich symbolizes that Hashem is always looking out for us.
While many of the reasons offered are connected to fish, the Magen Avraham (OC 583:5) writes that if necessary one may say tashlich at a river or stream that doesn’t contain fish.
The Maharil points out that it is forbidden to feed the fish (other than one’s own) on Yom Tov. Even carrying bread to feed them (outside of an Eruv) would be forbidden on Yom Tov. Rather, one should shake out their pockets as a symbolic act of throwing away one’s sins (Mateh Ephraim 598:4).
Whatever the reason, this time honoured minhag should serve to inspire us to do true teshuva. May we be forgiven and have our sins thrown into the depths of the sea.