The sages had a lot to say on proper attire during davening...
http://www.parshapages.com/Gemara/Shabbos/Proper%20Attire.doc"Hakon Likras Elokecha"
PREPARING FOR DAVENING WITH PROPER ATTIRE
The Sages teach that one should prepare oneself, both physically and spiritually, for prayer. One’s appearance and attire should be appropriate for an encounter with G-d; in addition, one should mentally and spiritually prepare for such a meeting. One should also limit certain activities before praying, in order to emphasize the centrality and significance of prayer in one’s daily religious routine.
The Gemara (Shabbos 10a): Rabba son of R. Huna put on stockings and prayed, quoting, hikon likrat elokecha “Prepare to meet your G-d” [Amos 4:12]. Rabba removed his cloak, clasped his hands and prayed, saying, “[I pray] like a slave before his master.”
Similarly, the Rambam (Hilchos Tefillah 5:5) writes: How should one prepare his clothes? First, one should adjust one’s clothes, and distinguish and beautify oneself, as it says “You should prostrate yourselves to His holiness in beauty.” One should not stand for Tefillah in an undergarment, with a bare head, or with bare feet, if the local custom is to appear before important people with shoes… The ways of the wise and their students is to pray while they are wrapped [atufin] in a tallis. The Kitzur Shulchan Aruch (12:1) adds that even one who prays alone in one’s house should dress appropriately.
The Poskim discuss whether one is merely required to dress as one would dress when meeting an important person, or whether one should do something extra, such as wearing a special belt, or gartel. This question arises from different understandings of the Talmud’s position regarding wearing a belt during prayer.
On the one hand, the Gemara (Berachos 24b) teaches: R. Huna said: If a man’s garment is girded round his waist he may recite the Shema. It has been taught similarly: If his garment, whether of cloth or of leather or of sacking, is girded round his waist, he may recite the Shema, but he may not say the Tefillah [Shemoneh Esreh] until he covers his chest. Apparently, the Gemara requires that one wear a belt in order to create a demarcation between one’s chest and one’s lower body. Seemingly, standing before G-d in prayer without this demarcation is viewed as inappropriate.
Another Gemara (Shabbos 10a), however, relates: R. Sheshes demurred: Is it any trouble to remove the girdle [before prayer]! Moreover, let him stand thus [ungirdled] and pray!? — Because it is said, “Prepare to meet your G-d.” According to this source, wearing a belt is an expression of “preparing to encounter G-d in prayer.”
The above two passages seem to disagree regarding whether one wears a belt in order to separate one’s upper and lower body during prayer, or whether it serves as an additional gesture in preparation for prayer.
Subsequently, the Rishonim debate whether one who does not ordinarily wear a belt must wear one for Shemoneh Esreh. The Ran (Shabbos 4a s.v. u-meha) and the Hagahos Maimoniyyot (Hilchos Tefillah 5:8) insist that even one who wears pants must wear a belt for the Shemoneh Esreh. On the other hand, Rabbeinu Yerucham writes that only one who is accustomed to wearing a belt daily must wear a belt for Shemoneh Esreh. Seemingly, these opinions disagree as to whether one’s attire for prayer should match, or should exceed, one’s usual standard.
R. Simcha of Vitry, a student of Rashi, records that Rashi would pray without a belt. When questioned about this practice, he responded that the Amoraim’s clothing was similar to our robes, without any demarcation between the upper and lower body. Nowadays, he explained, since it is customary to wear pants, no further separation is required. Rashi, apparently, viewed this halakhah in the context of the requirement to demarcate one’s chest from one’s lower body. Apparently, Rashi did not believe that one must wear a belt to fulfill hikon, preparing to greet G-d, either because he holds that separating between the upper and lower body fulfills this obligation, or because he believes that one fulfils the obligation through other preparations, such as netilat yadayim.
The Shulchan Aruch (91:1–2) cites both reasons for wearing a belt, in order to demarcate between one’s upper and lower body, and in order to prepare properly for Tefillah. The Magen Avraham and Mishnah Berurah, however, cite Rabbeinu Yerucham, who states that only one who normally wears a belt must wear one for Tefillah, but that one who does not ordinarily wear a belt need not wear one for Tefillah. The Mishnah Berurah does add, however, that there may still be a middat chasidut in wearing a belt. In addition, the Mishnah Berurah cites the Zohar, which states that it is appropriate to cover one’s head with a tallit during Shemoneh Esreh.
In modern times, some authorities have revisited this issue, questioning the propriety of wearing short sleeves, or even shorts, during prayer (see R. Ovadyah Yosef in Yechavveh Da’at 4:8). The Shulchan Aruch (90:5), for example, writes “One should not stand up for prayer… with exposed feet, if the accepted practice is to greet important people with shoes.” The Mishnah Berurah (91:11–13) points out that in hot countries where it is customary to stand before important people barefooted, one need not be concerned even if one is wearing “short clothing through which the legs are visible.” He also writes that one should not pray in sleepwear (pajamas), or other clothing that one would not wear to greet “important people.”
The concept can be better illustrated based on the last Brisker Rav in his sefer on Parshas Bereishis. The Brisker Rav takes note of the fact that Adam and Chava tried to hide from HaShem after eating from the eitz ha'da'as. Although HaShem expresses anger at them for being eating, He never accuses them of being so foolish as to think that they can hide from G-d. This implies that the hiding was actually an appropriate behavior under the circumstances (they had no clothes). The Brisker Rav explains this based on a Rashi in Berachos 24a that one can read Shema with a cloth wrapped around their waste (when the upper part of their body isn't covered), but cannot daven that way.
Rashi explains:
דלתפלה צריך הוא להראות את עצמו כעומד לפני המלך ולעמוד באימה, אבל ק"ש אינו מדבר לפני המלך.
Davening is essentially standing before HaShem, and when one does that they need a greater level of tznius and covering. When Adam and Chava ate from the tree and realized their nakedness, they at first were able to just cover the bottom half of their bodies. But when they heard HaShem's sound passing through the garden, they were obligated to hide themselves before G-d's presence to cover the top of their body as well, just as one would when davening to HaShem.
The black belt you refer to is called a gartel in Yiddish (similar to the English word "girdle").
A. Jewish law mandates that the "heart does not see the nakedness" when one recites the Shema and other prayers. This means that the upper body (more specifically the heart) be separated from the lower half, which has a coarser function. In ancient times, when common clothing consisted of a simple, loose robe, it was necessary to tie a belt around one's waist to insure that the nether region was out of view of the heart.
B. We read in Amos, "Prepare yourself toward the L‑rd your G‑d." 3 Our sages infer from here that one must dress himself up before facing his Maker in prayer. Part of this preparation is to gird oneself with a special belt. Hence the custom of wearing the gartel even though modern clothing ensures that that the "heart does not see the nakedness."
C. The gartel is reminiscent of the belt which the priests would wear during their service in the Holy Temple in Jerusalem. It is for this reason that many are particular to wear their gartel at elbow height, just as the priests of old did.