Data Engineering — the rebellious Cousin of Data Science

Multiple hats that data engineer has to wear in a day to day life!

1. Data Pipelines construction: The most prominent role is to make the batch-based(ETL) data pipelines by keeping the robustness, scalability, and the integrity of the pipelines in mind that carry the data from one place to another and keeping up the volume on which data is processed in a periodic cadence.Clearly, data engineers are expected to have a wide array of technical expertise. However, the job also requires critical thinking and the ability to solve problems creatively and knowledge of languages, frameworks, and tools (Python, Spark, Informatica)

2. Real-Time Streams: With advancements in technology, more and more prediction is done in real-time by deploying a model into the streaming pipeline. We also live increasingly in a world of real-time information and decision making. Building a streaming data pipeline (rather than batch-based) is yet another new set of skills that Data Engineers must implement, Kafka is a commonly used distributed data store optimized for ingesting and processing streaming data in real-time. A streaming platform needs to handle this constant influx of data, and process the data sequentially and incrementally. Other tools in this category are Kinesis, Pub/Sub, etc.

3. Business Domain Expertise: In most organizations, there’s a tremendous amount of legacy business information hidden in the company data. Without that domain knowledge, the information in the data is often missed, leading to data quality issues and without that one cannot build a robust warehouse which would lead to poor data-driven decision making.

4. Database Management: Extensive knowledge of database languages is required to do data engineering. Data Engineers must ensure that different databases are available to all users and functions without any hiccups. SQL and NoSQL are required skills here, along with advanced DBMS knowledge/skills as SQL is great for many structured data transformations and NoSQL for managing the semistructured data.

5. Optimization: The key skill here is not just to be able to build a data pipeline but to build one that is scalable and efficient. Higher-level skills are needed to design and build a data warehouse that can optimize the performance of queries, and when the data warehouse becomes very large, find new ways of making analyses perform better by tuning the queries.

6. Analytics/BI tools: One would think that this is the realm of the Data Scientists. It is for the most part. However, a Data Engineer must build a pipeline that supports data analysis and machine learning, so it helps to understand the terminology and outputs of the end-users. Data Engineers also need to use statistical modeling and tools (Powerbi, Tableau) on the job, for example, to measure the usage rate of data in a database, to build data quality monitors and end to end monitors to get a better picture of the source-target data comparison.

7. Cloud computing: Companies have data stored in various cloud warehouses (Redshift, Snowflake, Bigquery), buckets/lakes (S3, Blob storage, Cloud storage) and using multiple processing engines like (EMR, Dataproc, HDInsight) to process the data, for running their pipelines. A Data Engineer has to bring all these systems together to enable everyone in the company to use this data and should know how to manage the storage units, processing engines, and what should be the best hardware configurations to be used on the cloud to provide optimal Storage, speed and scalability for their platform according to their needs with the best pricing model.

8. Data Governance and Security: Data Engineers are not fully responsible for data governance, but they must assure systems are in place for data access and user control. They should have a background of data governance concepts and be sure that any tools and platforms they put in place support proper data governance.

9. Orchestration: Now that we can write a code to orchestrate the pipelines using Airflow it is becoming a new trend in the industry to expect that from a data engineer, they define direct acyclic graphs (DAGs). DAGs describes how to run a workflow. Workflows are designed as a DAG that groups tasks that are executed independently. The DAG keeps track of the relationships and dependencies between tasks, you can run DAGs schedule them to run at a specific time defined as a cron expression in the airflow.

Why Data Engineer role sometimes underestimated against Data scientists?

As we have seen, most of the time when a company succeeds all the credit solely goes to the data scientists as they are the ones who served the end dish. But in the end, we should all remember it is a team sport.
You can’t make a good pizza without its ingredients no matter how great are your cooking abilities. If your vegetables aren’t fresh, your cheese isn’t aged enough and your base isn’t moist enough, your pizza just won’t taste good.

So behind the scenes, data engineers are the ones who are keeping your data fresh, collecting, and maintaining years of data, and wrangling it into a suitable format to make it easily accessible to data scientists/other teams.

Great Data Science needs Great Data Engineering: Companies to hire a bunch of statisticians with higher degrees and are misguided to then roll out advanced machine learning solutions. Except, they fail to realize that there is a problem with the “DATA” itself. When you have a lot of scattered, uncleaned, and unlabeled data even the most advanced algorithms fail.
You need labelled, thoroughly cleaned, profiled, and analyzed data meticulously shaped into a suitable form for machine learning algorithms.

We tend to recognize only Data-based teams because of the new buzz words and forget about the domain teams who generate the data more or less without them Data engineers and Data scientists are nothing.
It’s a complete cycle and credit cannot be given to a single community.

Thus, it slowly becomes evident that machine learning is 60% Data Engineering which eventually starts clearing the fog on the importance of data engineers and how often their role is underestimated against Data Science.

We will continue to discuss more things around data domain on the upcoming part so stay tuned!