Complexity Drives Costs: A Look Inside BYOD and Azure Data Lakes

insightsoftware -
November 5, 2020

insightsoftware is the global provider of enterprise software solutions for the Office of the CFO to connect to & make sense of data in real time, driving financial intelligence across […]

Data lakes

If your company is using Microsoft Dynamics AX, you’ll be aware of the company’s shift to Microsoft Dynamics 365 Finance and Supply Chain Management (D365 F&SCM). Ostensibly, the new product represents Microsoft’s transition to a newer, more cloud-friendly ERP for midsized enterprises. It’s important to look beyond the surface, however, because there are some critical architectural changes that could dramatically affect how end-users get information out of the system.

Reporting will change in D365 F&SCM, and those changes could significantly increase complexity and total cost of ownership.

Video conference displayed on a laptop.

Before we dive into the details, it’s important to acknowledge that Microsoft is a platform company first and foremost. It sells a myriad of different software products, including a growing portfolio of software-as-a-service (SaaS) offerings. The company’s market power is based largely on its ability to promote the “stack”—that is, to position the entire suite of Microsoft products as a holistic solution to customer problems.

Although we think very highly of Microsoft and its product portfolio, that “stack” approach doesn’t always do the best job of addressing customer needs. Microsoft Dynamics AX customers who are considering a migration to D365 F&SCM should understand their options and plan accordingly. Those who take a default approach to the migration process may be in for an unpleasant surprise.

Let’s start with some background information.

The Data Security Problem: How We Got Here

You can extract data from relational databases, including Microsoft’s SQL Server using the SQL query language. In addition to reading data, however, you can also use the SQL language to insert, update, or delete records from a database. In the wrong hands, that can wreak havoc on your ERP data, resulting in lost information, data corruption, or theft. Even a well-intended expert can, on occasion, create a substantial problem by sending the wrong set of instructions to the database.

To enhance security, Microsoft has decided to restrict that kind of direct database access in D365 F&SCM and replace it with an abstraction layer comprised of something called “data entities”. Now, instead of making a direct SQL call to the database to get information, a report must query a kind of intermediary layer instead. As Microsoft envisions it, this will serve up all the information needed by a reporting engine, without allowing the inserts, updates, and deletions that could potentially corrupt the database.

Unfortunately, this approach has a significant downside—namely, it has the potential to be very slow. Virtually anyone who is running D365 F&SCM as their ERP system is processing enough data that they can run into substantial issues with reliability and performance when running reports using data entities. This has prompted Microsoft to come up with an alternate approach.

Bring Your Own Database (BYOD)

To address the performance challenges, Microsoft has devised an alternative approach it calls BYOD. That stands for “bring your own database,” and it refers to a model in which core ERP data are replicated to a separate standalone database used exclusively for reporting. That works reasonably well for traditional reporting functions. For more powerful, multidimensional OLAP-style reporting, however, it falls short.

OLAP reporting has traditionally relied on a data warehouse. Again, this entails creating a copy of the transactional data in the ERP system, but it also involves some preprocessing of data into so-called “cubes” so that you can retrieve aggregate totals and present them much faster. OLAP reporting based on a data warehouse model is a well-proven solution for companies with robust reporting requirements.

Unfortunately, Microsoft is not providing that kind of data warehouse solution out of the box. To get that level of flexibility and power, you would normally have to architect and build a data warehouse yourself. Fortunately, you can get it off the shelf and have the best of both worlds—power and flexibility—without the heavy lifting (more on that later).

Azure Data Lakes

This leads us to Microsoft’s apparent long-term strategy for D365 F&SCM reporting: Azure Data Lakes. That, in turn, brings us back to the point we made earlier—what is good for Microsoft isn’t necessarily always good for its customers. Azure Data Lakes are highly complex and designed with a different fundamental purpose in mind than financial and operational reporting.

For more on Azure Data Lakes, download this guide: “Diving into Data Lakes: Is Microsoft’s Modern Data Warehouse Architecture Right for Your Business?

Microsoft wants to drive the adoption of Azure Data Lakes, and it has a large customer base in need of financial and operational reporting. It’s only natural that the company would try to fit those two needs together. Unfortunately, the two are not an ideal match.

Data lakes are not a mature technology. They are designed for enormous volumes of information, including semi-structured and unstructured data. Unstructured data could include things like social media posts, online reviews, and comments recorded by a customer service rep, for example. In the context of a data lake, you can store unstructured information for later analysis and retrieval.

Structured data, including the kind of information that resides in your ERP system, can sit alongside unstructured data in a data lake. Traditional databases and data warehouses begin by defining what data should look like (that is, they must have a defined data model or “schema”). Data lakes move that step to the end of the process.

Consider the infrastructure required for this kind of mixed-use scenario. Data lakes are as much about governance, security, and data wrangling as they are about storage and retrieval. When you start to explore the complexities of Azure Data Lakes, you will quickly run into topics like data pipelines, conditional triggers, and machine learning algorithms. Those are highly relevant to a data scientist focused on big data analytics, but they’re far removed from the world of financial and operational reporting.

Azure Data Lakes are complicated. Complexity inevitably drives higher costs. That includes expensive and hard-to-find technical experts. In most companies, it also causes delays. If you need the IT department to write every new report, and every modification to an existing report thereafter, it becomes a bottleneck. At a time when agility matters most, management cannot get the information they need in time.

Today, data lakes are still a “bleeding-edge” technology, and they require significant technical knowledge and experience to understand their native system for unstructured storage. Because it’s still such new technology, there aren’t a lot of people in the labor market who have that expertise. That makes it even more expensive to rely on Microsoft’s Data Lakes approach.

One expert describes the deployment of Azure Data Lakes as “a multi-year marathon, not a sprint.” Upgrading your ERP system and supporting your financial and operational reporting needs shouldn’t be a marathon.

Microsoft’s strategy around reporting for D365 F&SCM is still in transition. The BYOD approach is serving as a short-term solution, but Microsoft clearly appears to be pushing customers toward data lakes as a long-term alternative. That may be good for Microsoft, but is it good for an organization that just needs a reliable reporting system?

Customers who follow Microsoft’s lead could very well end up investing heavily in architecture and report development, only to find they have to recreate a lot of that work down the road if the tech giant changes direction. It has happened before. In fact, Microsoft’s propensity to change direction is the reason why its Dynamics customers need to grapple with the question of data lakes at all.

A Turnkey Approach

There is another way to tackle your reporting needs in D365 F&SCM, using a solution that has been proven time and again. Jet Analytics from insightsoftware provides you with a turnkey solution to operational and financial reporting. Jet Analytics is built on a traditional data warehouse, designed to work with D365 F&SCM out of the box. It comes preconfigured for Microsoft Dynamics with a complete set of standard business entities that are ready to use. This makes the process of getting started with Jet Analytics remarkably fast and easy.

Jet Analytics was built from the ground up with finance and operational reporting in mind. We designed our toolset to enable end-users to create ad hoc and custom reports with minimal training, to remove the dependency on expensive technical resources.

Jet Analytics provides enterprise-grade capabilities at a lower total cost of ownership. Because we built our product on Microsoft technology and because it leverages the Azure platform, it offers a clear path to transition to data lakes, if and when you decide that it fits your strategy to do so. As the technology matures, it may eventually serve a broader range of customers with a wider variety of needs.

insightsoftware has helped thousands of companies achieve financial intelligence and operational excellence by making reporting easy. Our no-nonsense approach delivers fast, efficient, and effective reports that deliver high value and flexibility for end-users. If your business is striving to achieve greater speed and agility, insightsoftware can help you reach your goals.

If your organization is running D365 F&SCM and you are looking for a world-class reporting platform with a proven track record, download our Jet Solutions brochure, built for Microsoft Dynamics 365 to learn more about it.