Primus (in both instances) is not modifying causa, but is a predicative adjective going with dies.
Sentences where the action is modified by words like "first" or "last" generally work differently in Latin than they do in English. In English we'd ususally either use an adverb or an infinitival clause depending on a substantivized adjective:
- I did it first. (adverb)
- I was the first to do it. (substantivized adjective with infinitival clause)
In Latin, however, the ordinal generally appears as a predicative adjective agreeing with whatever it refers to:
- Primus id feci.
- hoc primus frater meus in Asia fecit "My brother was the first to do this in Asia" (Cicero, Pro Flacco 33)
Similarly in your sentence: Ille dies primus leti/malorum causa fuit can be translated as "That day first was the cause of ruin/evils" or "That day was the first to be the cause of ruin/evils", which should make it clear why "first" agrees with "day". (Of course it's perfectly valid to translate "That day was the first cause of ruin/evils", as well, but the Latin structure is less clear there.)