Data Literacy Assessment : How to use them to start or scale data literacy?

In the past, training was the top choice for getting started on data literacy. But since 2022, data shows 82%+ of people responsible for data literacy are starting with data literacy assessments and a focus on data culture.

Why the change? Here are three primary reasons:

1. The individual(s) responsible for data literacy are accountable for success metrics that are connected to real business goals.
2. A data literacy strategy has become real. More than simply implementing new data tools or software.
3. The goal of data literacy is about developing real competency and less about training completions or a checklist of skills.

What do Data Literacy leaders and key stakeholders believe will work?

1. Before they invest in training, they want to be clear on which competencies and skills they need, why they need them, and how they connect to their business goals.
2. They want competencies that drive a data culture and mindset. Not just test knowledge of tools, stats, and charts.
3. They want assessments that baseline their data literacy assessment model. Not pre-training tests.

How to get started with Data Literacy Assessments?

Whether you buy data literacy assessments or create your own, here are the basic steps for getting started with data literacy assessments. The steps are sequential, but each one is dependent on choices you make in previous steps.

1. State your data literacy assessment goal
2. Define competencies (also sometime called desired outcomes or behaviors)
3. Define and prioritize skills by roles or functions
4. Choose assessment types

1. State your goal

Stating the purpose of the data literacy assessment makes it clear how it will be used. To draft your goals, try using the “so that” statement to communicate the “why”.

Here is an example:

Goal: The assessment will measure the competency and skill gaps that exist within specific groups, categories, or roles of employees.

2. Define Competencies

We all have a tendency to use the word “skill” for everything. When you start with data literacy assessments it helps to be more intentional about definitions.

Competencies are the desired outcomes, or behaviors, that clearly describe what people should be able to do and implies some level of ability or effectiveness. 

Competencies are very different from skills. They are not tasks.

Competence results when we use a combination of skills at the same time in specific situations. For data literacy, competencies should be specific to roles in an organization and in the context of doing data analytics.

You can find data literacy competency models just by doing a little research. How do you know if it is any good? Here are a few tips for evaluating competencies:

Does the competency model: 

• Describe what you want people to be able to do on the job?
• Describe what it looks like when they are doing it well?
• Is the description stated as objectively as possible?
• Does it also describe the effect or impact the actions should have? A result or effect on other people.

If you cannot answer yes to all four, the model is probably generic or high level. This leads to training that is generic and lacks relevance. It also results in weak competency measures.

For example, many data literacy competency models list things like curiosity, problem solving, communication, etc and have a short sentence about each of them. These are important categories. But they lack clear descriptions of what “ability” looks like. And they do not differentiate what competent looks like for different roles in your organization. 

Because we, at Aryng, have spent years working together with data scientists, analysts, data engineers, and business people on real analytics projects, we have been able to document specific, desired outcomes that lead to business impact. We have put these into our Data Literacy Competency Bank.

For example, here are two key competencies that we have under the category of Data-driven Mindset:

“I assume positive intent when being questioned or when questioning others about data, insights, and recommendations.”

“I understand data driven recommendations and the impact my actions should have when implemented.

For data literacy, it is common to see different roles have 8-10 key outcomes that communicate the highest priorities.

3. Define skills

When an organization creates their own competency model, it is much easier to see which skills are most important to the organization. Skills map to competencies. They are not “one-offs”.

You can come up with a list of skills by doing some research. But again, it is someone else’s list. Skills without a customized competency model will lack relevance to business goals and impact.

There are 65 or more skills in data literacy. Your competency model helps you prioritize skills. Get further input from leaders, managers, and people who are responsible for developing data literacy. 

If you have the time and expertise, you can evaluate job descriptions for key skills. However, most organizations lack current JDs or have not adapted them for data literacy. 

Skills descriptions should be specific to different roles or personas and describe how it is applied in the context of data analytics. Skill descriptions are used to write data literacy assessment questions. If your skill definitions are vague, your assessment data will be also.

For example, an organization shows “Listening” as a data literacy skill and defines it as “nodding and asking follow up questions.” This definition lacks specific on how listening is applied in data analytics or how someone in marketing or operations would use it. How would you assess “nodding”?

Finally, we see many models that have skills tagged to “levels” like beginner, advanced, expert. Do you need to define levels to get started? Generally speaking, no.Skill levels are not easy to define. So they end up being a relatively meaningless label. Definitions like, “needs some help, can do it with some coaching, can teach others”, do not describe ability or expertise. 

It is better to concentrate on well crafted competencies that map to business goals, and skill descriptions that map to competencies. 

4. Types of assessments

This step involves choosing objective tests or subjective surveys, or both. Your decision depends on the previous steps – your goal and what you will assess.

It is generally believed that objective tests are far more accurate than subjective surveys. Not always! Some expertise may be needed in helping you choose the best type of assessments, to make sure the assessment questions are written properly, or to do some level of assessment validation. 

In summary, starting with assessments before you buy training gives you the opportunity to take a customized assessment approach that fits your needs and will align data literacy to strategic business goals.

1. State your goal.
2. Define competencies (aka desired outcomes or behaviors) that align with your business goals and culture of data driven decision making. Map competencies to your organization’s roles or functions. Or use them to define your personas.
3. Choose skills that align to your competencies versus a generic list or pathways.
4. Draft assessments that test relevant skills and tie back to competency descriptions.

If you start with an existing competency, skill, and assessment question bank these four steps should take no longer than a few weeks to fully customize. If you start from scratch, it may take several months. Make sure you have the resources, expertise, and time.

Finally, two mistakes we see most often are:

• Defining competencies or skills before you define personas
• Personas that are based on how the role handles data (aka data consumer and data producer)

Either one leads to missing competencies and leads to weak assessments. 

Assessments are the most valuable way to measure your data culture and drive maturity when your competencies and skills align to your expectations for a data-driven mindset and decision-making process that leads to business results.

For more information on getting started with assessments, see our Data Literacy Playbook at aryng.com.