Where is the perfect diet?

Every couple of years there is a new popular diet, nowadays usually people start talking about it online, celebrities start following it and keeps growing until a new diet comes along. Recently intermittent fasting has gained a lot of popularity, but in the near a future another diet will start gaining traction and surpass intermittent fasting. But why is this? Why don’t we have a clear answer as we have in other scientific fields?
One of the main reasons is that everybody is different, not everybody has the same dietary needs, and some people suffer from illnesses that may require or reject different macronutrients, a clear example of this is that some people with immunodeficient issues benefit greatly from the so-called carnivore diet, where meat is the almost sole provider of calories. But it is clear that this is not the best alternative for everybody, as usual, extremes are not the solution, and the long term effect are unknown.
Another factor that makes definitive research difficult is that it is usually based on surveys, where people are asked to change their dietary habits and report on several relevant health metrics, which is problematic in the sense that people are not perfect, they try to do what they are asked but can not always deliver, so the data is not always perfect. Related to this we have the issue of long term effects because humans live for a long time compared to other animals, and even though a diet may provide benefits in the short term, a thorough followup on the study participants must be carried out to discover the long term effects of such diet. All this is not easy to do because of human imperfection and because of resources limitation.
Industry-funded research is also a source of misinformation because contradicting research is generated to back the health benefits of an industry, an example of this are the studies carried out by the alcohol or soft drink industry to create doubts on the health risk associated with such products. They are able to manufacture studies where a correlation is found between two things even though the methodology was designed from the start to get to the desired result.
But in my opinion, the future seems bright in this topic, as with many others, machine learning, the increasing generation of relevant data and the decreasing price in DNA sequencing may uncover the perfect diet for each one of us, the one that maximises our objectives, either living the longest or living with the most energy or a combination of both. But in the meantime, we must try to follow scientific consensus as it is the best we have at the moment and that consist of a low-sugar, low-fat balanced diet combined with moderate exercise.