I have mixed feelings about this. The issue is that a lot of larger corporations still use GUI-based ETL solutions. These GUI-based systems also provide an all-in-one DE platform solution and they have not stopped adding features/value to the product.
Migrating them off to a code-based system is going to be an incredible challenge, and the value is simply not there. Data Engineering as a concept is less of a "code" based solution, rather a platform-based solution. Code is supposed to be modular. Even the code-based solutions are more like simple, plain-text configuration files rather than code.
This is the same path I've been on in discovering AI works really well with addressing variations in data as long as there is a structured system.
So glad other people are thinking about this too! Looking forward to your future writings!
I have mixed feelings about this. The issue is that a lot of larger corporations still use GUI-based ETL solutions. These GUI-based systems also provide an all-in-one DE platform solution and they have not stopped adding features/value to the product.
Migrating them off to a code-based system is going to be an incredible challenge, and the value is simply not there. Data Engineering as a concept is less of a "code" based solution, rather a platform-based solution. Code is supposed to be modular. Even the code-based solutions are more like simple, plain-text configuration files rather than code.
This post is about builders, not legacy owners. It's hard to argue to replace legacy, if it was easy, it wouldn't be legacy.