Breaking Down Information Silos: A Sensible Framework from the Subject – Atlan

Information silos are a serious roadblock to efficient decision-making, forcing groups to spend extra time discovering and getting ready information relatively than analyzing it. In truth, IDC stories that information professionals dedicate almost 80% of their time to data-related duties, leaving little room for precise insights. On this article, we discover an actionable framework designed to encourage a extra linked method to information administration, highlighting real-world examples and sensible methods.

What are information silos and their impression?

At their core, information silos emerge when info is remoted inside two major mechanisms: folks and know-how. 

Departmental boundaries and organizational constructions naturally create obstacles to info sharing. Moreover, the rise of specialised instruments and platforms means essential enterprise information typically stays trapped in disconnected techniques.

And the impression of those silos? These are a few of the widespread points prospects have shared with us. 

• Time and Effectivity Impacts: Information silos pressure groups to spend days or even weeks on easy information requests, and may trigger widespread duplication of labor as completely different teams unknowingly sort out the identical tasks. As information volumes develop, backlogs develop sooner than groups are ready course of, creating mounting effectivity issues.
• Information High quality and Belief Points: A number of variations of the identical information emerged throughout completely different techniques, making it not possible to find out which model was correct and up-to-date. Eroding belief in information and KPIs all through the group. This creates extra effort to confirm information and hesitation to even use it. 
• Scale and Progress Issues: As firms expanded, centralized groups couldn’t sustain with growing information calls for, resulting in a proliferation of uncoordinated instruments and techniques. Groups started working in isolation, and new applied sciences had been acquired with out correct adoption planning, creating an more and more fragmented information panorama.
• Discovery and Entry Challenges: Precious information property grew to become successfully invisible inside organizations as groups had no centralized solution to discover or entry current information. Customers remained unaware of related information that would assist them, whereas information house owners had no visibility into who was utilizing their information or the way it was being utilized.
• Useful resource and Value Implications: Organizations waste cash on duplicate information storage, redundant instruments, and underutilized techniques whereas inefficiently deploying engineering sources. The hidden prices of knowledge silos manifested in each direct bills and alternative prices from inefficient useful resource allocation.

Regardless of these points, breaking down information obstacles is achievable with the proper technique. 

Fixing for information silos requires a twin technique with a 6-part framework

The identical two elements that trigger information silos to emerge, folks and know-how, additionally illuminate the technique for dismantling them. Analyzing every space reveals a path ahead for organizations searching for to interrupt down information obstacles.

Let’s begin with the human component. At its core, the answer begins with fostering the proper tradition. When groups perceive how information impacts enterprise choices and their position in proudly owning that information, a mindset emerges the place info is seen as a shared asset relatively than departmental property. 

Creating this tradition would require funding and organizational change. It means educating groups on information literacy, securing cross-functional dedication to collaboration, and gaining sturdy buy-in from management. 

Whereas these modifications are difficult and take time to implement, the result is transformative: a tradition the place collaboration thrives as a result of everybody shares and has entry to the information they should make enterprise choices.

Addressing know-how is simply as essential as tackling organizational silos. Within the trendy information panorama, info property are sometimes scattered throughout varied techniques and platforms. Gross sales groups could depend on Salesforce, whereas advertising groups use HubSpot. Information will be unfold throughout a number of cloud suppliers like Azure and AWS, and even dispersed globally.

With out correct administration, this variety of instruments can create chaos. Efficient information administration requires extra than simply connecting disparate techniques. It calls for a complete method that ensures information high quality, maintains consistency, and preserves important context. This implies implementing processes and governance to make sure information is correct, up-to-date, and adheres to organizational requirements. It includes establishing clear possession and accountability for every dataset. Crucially, it additionally necessitates capturing and sharing essential metadata that gives perception into the information’s origins, that means, and lineage. When executed proper, it permits groups throughout the group to know, belief, and successfully make the most of the data they uncover. 

Whereas there’s no common guidelines for fixing information silos, we’ve developed a complete 6-part framework—a strong, adaptable roadmap that organizations can implement and grasp. The success tales of visionary firms like Autodesk, ContentSquare, Kiwi.com, Nasdaq, Porto, and North help this method and showcase how the framework can navigate distinctive challenges and democratize information throughout completely different organizational contexts.

1. Area Empowerment with a Information Middle of Excellence

Empower groups to be instantly accountable for their information by means of a robust centralized information workforce that establishes the inspiration, requirements, and instruments for trusted information merchandise.

At Autodesk, by early 2021, their central Analytics Information Platform workforce confronted an amazing backlog of knowledge ingestion requests that exceeded all beforehand ingested information within the platform’s historical past. They solved this problem by implementing a collaborative area possession mannequin, one the place groups had been empowered to publish and handle their information merchandise whereas working inside a shared governance framework. This ensured interoperability and accessibility throughout the group. Their platform workforce established widespread requirements, instruments, and processes that allowed 60 area groups to independently construct information merchandise whereas making them discoverable and usable by others by means of a central catalog. This balanced method led to the profitable supply of 45 new use circumstances in simply two years whereas guaranteeing information remained accessible and helpful throughout workforce boundaries.

2. Clear Governance Construction

Present the framework for the way information is managed, documented, and shared throughout a company. 

Main firms have applied numerous but efficient governance frameworks to handle their information property: 

• Contentsquare applied a hybrid possession mannequin the place their Data Techniques Division (ISD) maintained system-level management whereas enterprise items retained information possession, complemented by information compliance ambassadors embedded throughout departments to make sure constant requirements throughout acquisitions.
• Porto established a tiered governance mannequin, designating property as both “Full Governance” (requiring full documentation, classification, high quality checks, and outlined possession) or “Simplified Governance” (needing solely primary lineage and cataloging), permitting their small five-person workforce to effectively handle over 1 million information property. 
• At Nasdaq, they advanced from a centralized reporting construction to a federated mannequin with specialised groups. The Platform Workforce maintained core know-how, an Financial Analysis workforce centered on information science, and embedded analysts established inside enterprise items, all working inside outlined engagement protocols. 
• Kiwi.com took a product-centric method to governance, protecting possession, documentation, high quality, structure, safety, and processes, which helped them scale back engineering workload by 53% whereas growing information consumer satisfaction by 20%. 

These diversified approaches showcase that efficient governance constructions can take completely different varieties whereas attaining the identical objective: creating clear, scalable frameworks for managing and sharing information throughout complicated organizations.

3. Constructing Belief By Requirements

Set up constant practices for the way information is created, maintained, and shared inside a company. 

Kiwi.com’s huge information panorama of 100 Postgres databases and tens of 1000’s of tables resulted in overwhelming search outcomes for analysts, with easy phrases like “Vacation spot” returning over 200,000 entries. They tackled this by creating a governance framework for information merchandise and establishing requirements round possession, documentation, high quality, structure, safety, and processes. This reworked their method from merely storing information to curating 58 fastidiously managed, top-tier information merchandise that met strict standards for reliability and accessibility. Every information product now requires technical and product-level possession, complete documentation, high quality monitoring by means of their observability platform, and formal information contracts specifying SLAs and SLOs between producers and shoppers. Their central engineering workforce noticed a 53% discount in workload as groups grew to become extra self-sufficient, information consumer satisfaction elevated by 20%, they usually efficiently onboarded over 20 groups to responsibly share and use information throughout the group. This standardized method not solely improved effectivity but additionally constructed confidence of their information, enabling groups to make sooner, extra knowledgeable choices with dependable information merchandise.

4. Unified Discovery Layer

Present a single place to search out, perceive, and entry information throughout a company. 

At Nasdaq, groups would go to a number of teams concurrently to get information solutions, with Michael Weiss noting “In case you had been a enterprise unit, you can go to one in every of 4 groups to attempt to get a solution. As a matter of truth, they might go to all 4 groups to see who would win first.” Their survey revealed that energy customers spent one-third of their time simply attempting to know the context round information they already had entry to. They reworked this by implementing Atlan as their unified metadata layer, which customers described as “having Google for our information.” This allowed groups to rapidly uncover current information property, perceive their context, and ask the proper questions, relatively than duplicating work throughout a number of groups.

5. Automated Governance

Use know-how to constantly apply information administration guidelines and insurance policies at scale. 

At Porto, a five-person information governance workforce was manually managing over 1 million information property, spending important time on tedious duties like updating classifications and possession in spreadsheets. They reworked this by implementing Atlan to automate their governance processes. The automation categorised property as both “Full” or “Simplified” governance primarily based on preset guidelines, mechanically assigned possession primarily based on information domains, and recognized potential PII fields by means of sample matching. This resulted in a 40% discount in time spent on guide governance duties, permitting the workforce to concentrate on extra strategic work. As Danrlei Alves famous, “With the time we saved, we’re growing the scope of labor we have now, and tasks we do. For the guide actions we nonetheless have, we’re discovering a solution to automate them, too.”

6. Related Instruments & Processes 

Create a seamless circulate of knowledge work throughout completely different platforms and groups. 

At North, their information workforce was fighting inefficient situation decision and duplicate work throughout their Snowflake and Sigma environments. Workforce members would obtain tickets with out clear references to particular information property, typically resulting in a number of engineers unknowingly engaged on the identical points. By integrating Atlan’s Chrome Extension with Jira and Slack, they enabled customers to lift points instantly from Sigma with automated hyperlinks to particular property. As Daniel Dowdy defined, “Eliminating duplicate work, or eliminating your engineers engaged on the identical undertaking with out realizing about it? These effectivity positive aspects are large, and the financial savings add up actually rapidly.” The combination additionally supplied engineers with a complete historical past of all tickets and conversations regarding particular information property, serving to them higher perceive and stop related previous points.

Your Path Ahead: From Framework to Implementation

The challenges posed by information silos are multi-faceted, however solvable by means of a strategic, coordinated method. As we’ve seen, siloed information stems from two core points:

1. Fragmented possession and accountability
2. Lack of unified processes and tooling for discovery, governance and entry

To systematically break these obstacles down, organizations ought to concentrate on constructing six key capabilities:

1. Set up clear area possession to empower groups
2. Implement sturdy governance constructions and processes
3. Construct belief by means of constant requirements and information contracts
4. Create unified discovery and exploration layers
5. Automate governance duties to make sure scalability
6. Join instruments and workflows to allow seamless collaboration

The facility of this framework has been confirmed out by pioneering information organizations. North American Bancard struggled with scattered information and guide governance holding them again—making use of automated governance with Atlan drove a 700% improve in managed information property. Autodesk handled information trapped in workforce silos, however by rolling out standardized processes and a central discovery layer, they broke obstacles between domains. Contentsquare lacked dependable entry to trusted information, by means of structuring governance and possession, they unlocked fluid, safe information sharing.

Every of those successes validates how a scientific method to dismantling information silos can speed up a company’s progress towards changing into really data-driven. Whereas the journey takes dedication, the vacation spot is evident: empowered groups seamlessly accessing and enriching a shared pool of trusted information to drive innovation.

Able to discover how your group can operationalize this framework to attain data-driven transformation? We’d love that will help you map out the proper path ahead. Ebook a demo with our workforce to see how Atlan might help you break down silos and democratize your information.