The issue is that the letters are our earliest Christian writings and while some of the gospel sayings probably do go back to Jesus, the gospels were still composed decades later and by anonymous writers. Paul at least did interact with the disciple Peter and Jesus’ brother James.