The Rema (OC 583:2) writes that many have the custom to avoid eating nuts on Rosh Hashana. One reason for this is because אגוז is the same gematria as חט, sin (minus the א – the way it is spelled in Talmud Yerushalmi). Similar to the various simanim we eat on Rosh Hashana, we want to remind – and inspire - ourselves to do teshuva as one eats – or avoids - these special foods (Matei Ephraim 583:2). Thus, many avoid bitter foods.
While many avoid nuts throughout the aseres yemei teshuva, there doesn’t seem to be any source for this.
R’ Shmuel Kamenetsky (quoted in Kovetz Halachos: Yomim Noraim) holds that while one may eat food with nuts mixed or baked in, they should not be recognizable. (Thus, smooth peanut butter may be consumed, though chunky peanut butter should be avoided.)
While this is an ashkenazic minhag, some Poskim point out that the Maharam Mi’Rottenburg used to eat nuts on Rosh Hashana.The Mishna Berura (583:5) writes that while some are particular not to pickle their fish in bitter brine as a good siman, it is more important to be careful not to get angry with another during this time, as a good siman! Similarly, the Beis Yisroel (5th Gerrer Rebbe) points out that while many avoid nuts because it shares the same gematria as חטא, we mustn’t forget that חטא also shares the same gematria as חטא. Let’s remember these simanim for what they are, and use them to further inspire us to teshuva.