5.1 C
United States of America
Tuesday, November 26, 2024

Quick is sluggish, sluggish is quick – rethinking Our Knowledge Engineering Course of | Weblog | bol.com


Rethinking Our Knowledge Engineering Course of

Whenever you’re beginning a brand new crew, you are usually confronted with a vital dilemma: Do you stick along with your present manner of working to rise up and operating shortly, promising your self to do the refactoring later? Or do you are taking the time to rethink your method from the bottom up?

We encountered this dilemma in April 2023 once we launched a brand new information science crew targeted on forecasting inside bol’s capability steering product crew. Throughout the crew, we regularly joked that “there’s nothing as everlasting as a brief resolution,” as a result of rushed implementations usually result in long-term complications.These fast fixes are likely to grow to be everlasting as fixing them later requires vital effort, and there are all the time extra quick points demanding consideration. This time, we have been decided to do issues correctly from the beginning.

Recognising the potential pitfalls of sticking to our established manner of working, we determined to rethink our method. Initially we noticed a possibility to leverage our present expertise stack. Nevertheless, it shortly grew to become clear that our processes, structure, and general method wanted an overhaul.

To navigate this transition successfully, we recognised the significance of laying a powerful groundwork earlier than diving into quick options. Our focus was not simply on fast wins however on making certain that our information engineering practices may sustainably help our information science crew’s long-term objectives and that we may ramp up successfully. This strategic method allowed us to handle underlying points and create a extra resilient and scalable infrastructure. As we shifted our consideration from fast implementation to constructing a stable basis, we may higher leverage our expertise stack and optimize our processes for future success.

We adopted the mantra of “Quick is sluggish, sluggish is quick.”: speeding into options with out addressing underlying points can hinder long-term progress. So, we prioritised constructing a stable basis for our information engineering practices, benefiting our information science workflows.

Our Journey: Rethinking and Restructuring

Within the following sections, I’m going to take you alongside our journey of rethinking and restructuring our information engineering processes. We’ll discover how we:

  • Leveraged Apache Airflow to orchestrate and handle our information workflows, simplifying advanced processes and making certain easy operations.
  • Discovered from previous experiences to establish and eradicate inefficiencies and redundancies that have been holding us again.
  • Adopted a layered method to information engineering, which streamlined our operations and considerably enhanced our means to iterate shortly.
  • Embraced monotasking in our workflows, bettering readability, maintainability, and reusability of our processes.
  • Aligned our code construction with our information construction, making a extra cohesive and environment friendly system that mirrored the best way our information flows.

By the top of this journey, you’ll see how our dedication to doing issues the suitable manner from the beginning has set us up for long-term success. Whether or not you’re dealing with comparable challenges or trying to refine your personal information engineering practices, I hope our experiences and insights will present invaluable classes and inspiration.

Float

We rely closely on Apache Airflow for job orchestration. In Airflow, workflows are represented as Directed Acyclic Graphs (DAGs), with steps progressing in a single path. When explaining Airflow to non-technical stakeholders, we regularly use the analogy of cooking recipes.

Related Articles

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Latest Articles