Analytics Workflow: A Technical Approach (Christopher Penn)

Christopher Penn


Analytics workflow is a critical part of data-driven decision making. It involves collecting, transforming, and analyzing data from multiple sources, and then presenting the output for stakeholders to make decisions.

This article will discuss how to set up an analytics workflow, particularly for addressing the problem of having to pull data from multiple sources. We will focus on a technical approach, which is best suited for technically savvy users.

Identifying data sources

The first step of setting up an analytics workflow is to identify the data sources and their export formats. The two most common formats for data exports are JSON and CSV.

  • JSON stands for JavaScript Object Notation and is an unstructured data format that allows for flexible data exports.

  • CSV stands for Comma Separated Values and is used for rectangular data.

Additional formats such as RSS, XML, and SQL may also be available in some services. It is important to catalog the data sources and their export formats so that you know what you need to connect to each of them.

User story gathering

The next step in the analytics workflow process is to do user story gathering. This involves asking stakeholders what data they need, why they need it, and what their intended use case is.

To do this, we recommend that they use the user story format:

“As a [user], I need to [action] so that [outcome].”

For example, a social media marketer may say,

“As a social media marketer, I need to understand the engagement rates of my audience on the various social channels so that I know where to allocate my social media spend and personnel.”

From this user story, we know that we are looking for engagement rates, audience sizes, and different social media channels.

Connecting to Data Sources

Once the data sources and user stories have been identified, the next step is to connect to the data sources and extract the data. This is typically done using a programming language such as R, although some people may use Python or off-the-shelf products such as Tableau.

The goal is to ingest all the data on a regular and frequent basis with automation and then perform ETL (Extract, Transform, Load) processes. During this step, the data should be transformed by changing date formats, standardizing numbers, cleaning up text, and doing feature engineering to infer data that might not be in the original data source.

Storing Data

Once the data has been transformed, it should be loaded into some form of storage for further analysis. This could be a service such as Google BigQuery or a local piece of software such as MariaDB.

Analyzing Data

The next step is to do the analysis itself. This involves taking the user stories, converting them into code, processing the data, and then getting it ready for output.

During this step, feature engineering should be done as well as running fancy algorithms such as regression analysis or driver analysis.

Outputting Data

The last step of the analytics workflow is to output the data. This should be done in a way that meets the needs of the stakeholders. For example, if a social media manager is looking to allocate time or resources, the output should be a bar graph with engagement rates per channel so that the manager can quickly make a decision.

Conclusion

Analytics workflow is an important part of data-driven decision making. This article discussed how to set up an analytics workflow, particularly for addressing the problem of having to pull data from multiple sources. The process involves identifying the data sources, doing user story gathering, connecting to the data sources, storing the data, analyzing the data, and outputting the data. This approach is best suited for technically savvy users and involves writing code, performing ETL processes, and using feature engineering.


Christopher Penn @cspenn
Co-founder, Trust Insights


Tell Us Below:

How do you set up your analytics process?

2 Likes

When I start off my clients, we have a strategy meeting and we start with goals. It all starts with goals. Then we look at what they’ve been tracking previously, and what they aren’t tracking (most often, they are barely tracking anything!). Then we make a plan as to what are the most important metrics for their business growth. We look at analytics together each month, and adjust if we can’t find something, or need to dig deeper.