I will start with my comments.
Quotations follow after the line.
As you wrote, diligere is about esteem, amare is about passion.
With this in mind, I would not consider diligere colder than amare.
A husband can love his wife in an erotic way (amare) and he should probably let her know about that, but it is also important that he values her as a person and is proud to share his life with a good human being (diligere).
Both feelings are important to the relationship, and both have their appropriate times, but I don't see diligere as a lesser form of love.
Similarly, I would say that there can be amor without dilectio and vice versa.
Suppose a woman lives with a man whom she finds very interesting and is proud to help him with his important career, but she feels no physical attraction.
There is dilectio without amor.
Suppose the same woman has a physically attractive lover who always makes her laugh and feel good but who doesn't do anything valuable with his life.
The amor can be fierce, but there might not be any dilectio.
However, I would argue that true and deep amor (whatever that means) requires dilectio.
(If you ask me whether there can be feeling A without feeling B, the answer is almost always yes, as long as A and B are not exactly identical. But this has nothing to do with Latin.)
Amare indeed refers to all kinds of love.
To refer to sexual love as opposed to love between friends, I would consider words like calor, cupido, cupiditas, ardor, and libido.
These are all nouns; I don't know a good verb for this purpose.
Everything so far is based on my interpretation of these two words based on dictionaries.
L&S compares the two verbs in the entry amo.
On one hand, "Amare in ccntr. with diligere, as stronger, more affectionate", but on the other hand "But diligere, as indicative of esteem, is more emph. than amare, which denotes an instinctive or affectionate love".
There are several examples of the two verbs used in the same context.
It is surprisingly common in Cicero, many of which are not mentioned in the linked dictionary entry.
Here are some examples:
Cic. Amic. 100:
… exardescit sive amor sive amicitia; utrumque enim ductum est ab amando, Amare autem nihil est aliud nisi eum ipsum diligere quem ames, nulla indigentia, nulla utilitate quaesita. quae tamen ipsa ecflorescit ex amicitia, etiam si tu eam minus secutus sis.
Falconer's translation:
… love or friendship leaps into flame; for both words are derived from a word meaning “to love.” But love is nothing other than the great esteem and affection felt for him who inspires that sentiment, and it is not sought because of material need or for the sake of material gain. Nevertheless even this blossoms forth from friendship, although you did not make it your aim.
If I read this correctly, Cicero says that to love is to esteem highly.
(I would rather say that deep love requires esteem, but I don't want to challenge Cicero's statement here.)
Here is an example of the two verbs used together:
Cic. ad Brut. 1.1.1:
Clodius […] valde me diligit vel, ut ἐμφατικώτερον dicam, valde me amat.
My rough translation:
Clodius thinks highly of me, or, to put it more emphatically, he really loves me.
This gives the impression that amare is stronger than diligere.
I think diligere means roughly the same thing in this quote and the previous one, but amare does not.
Here it looks more like "to like a lot" than "to feel love".
(Compare "I love ice cream!" with "I love you!" to see the same effect in English.)
The same comparison also appears in a letter of his: eum a me non diligi solum, verum etiam amari.
Cic. Ver. 2.4.51:
Homo nobilis, qui a suis amari et diligi vellet…
Yonge's translation:
That noble man, as one who wished to be loved and esteemed by his fellow citizens…
This passage indicates that amare and diligere are complementary instead of one containing the other.
I have only looked at Cicero, and the meaning and especially the difference of amare and diligere is depends much on context.
There is an overall tendency for amare to mean love and diligere esteem, but this is only a rule of thumb.
Comparison of these two verbs in Cicero alone is a broad subject and worth separate study.
