QA in a Big Data and AI World

So, what is Big Data and how does it relate to Artificial Intelligence you may ask…

Well, Big data is a term used to describe the enormous amounts of data being generated in our organizations by the latest and greatest applications that are being developed daily. The data generated by such applications, including many Artificial Intelligence programs, is growing at an exponential rate. For example, the Google Maps car generates 1GB of data per second!

Previously, it was understood that we only needed to hold onto the data for short periods of time where after it could then be sent off to the depths of the formidable archives. Since then we’ve discovered that within the vast basements of said archives lies the treasure trove of information that can be used to help us better understand our customers and what they like or don’t like. An example of this would be targeted marketing, whereby advertisers are now personalizing adverts according to customers purchase history.

And so, with all this new found treasure, began the era of Data Science involving predictability studies, algorithms, statistical modelling and data research feeding a constant need to know more about our customers. This is incredibly valuable information, but we must ensure that the data is accurate otherwise it’s useless and ineffective in its purpose.

How do we do that?

In order to ensure data accuracy, we need to understand the challenges that lie within. 3 of those issues are…storage, access and analysis. For this, we have the 4 V’s – Volume, Velocity, Variety and Veracity. The data is enormous in size (Volume) and it needs to be able to move at speed (Velocity), also it varies in its form (Variety) and sometimes there are unknowns (Veracity) for which we must be prepared.

In order to assure the quality of the data, there are different phases of testing which can be used to provide clarity and confidence in the data:

Phase 1 – Data Staging Validation

Data is gathered from various sources into Hadoop Distributed Filing System. Data is then compared to the source data and verified to be correct.

Phase 2 – MapReduce Validation

This consists of two methods. The Map method which filters and sorts the data, and the Reduce method which is a summary operation which involves counting the data. Lastly, in this phase verification of business logic takes place.

Phase 3 – Output Validation

Output data files are generated and moved to the Enterprise Data Warehouse where the data will be extracted and reported on using BI Tools.

Due to the sheer volume and sensitivity of the data other extremely important forms of testing would be Performance, Failover and Security testing. But that’s a whole different can of worms!

Artificial Intelligence

Who remembers Knight Rider? The 1982 all American TV series that took us into the world of a crime-fighting hero, Michael Knight, with a sidekick like no other – a self-driving, bulletproof, fireproof car that could speak called KITT. Perhaps, the creators of KITT had an inkling into what lay ahead in the developing world of AI.

Artificial Intelligence could be defined as:

“Intelligence demonstrated by machines, in contrast to the natural intelligence displayed by humans.”

There are many facets of AI but there are 4 that should sound familiar to most of us:

  • Robotics – the use of engineering and science for information processing
  • Machine Learning – use of algorithms relying on patterns and inference
  • Natural Language Processing – the processing of human languages by computers
  • Artificial General Intelligence – problem solving, reasoning and consciousness

Artificial General Intelligence is probably one of the most researched and funded forms of AI and illustrates the ability of machines to perform the full range of human cognitive abilities.

What testing is done to confirm this, so called, human-level AI?

  • The Turing Test – Machine & Human Vs. Human
  • The Coffee Test – Robot walks into a home and makes coffee
  • The Student Test – Register as a student and obtain a degree
  • The Flat Pack Furniture Test – Unpack and assemble furniture
  • The Mirror Test – Distinguish real vs reflection

The impact of AI on all of us today is far greater than we sometimes realize. In any given day we use, on average, close to 10 forms of sophisticated AI. Some examples are Apple’s Siri, Amazon’s Alexa, Google Translate, Netflix, Google Maps, Uber, Instagram, Pinterest etc.

It seems that the dreaded concept of Artificial Intelligence is already very much part of our daily lives. Best we get used to it!

Below is an interesting video of the world’s first Robot citizen called Sophia. Enjoy!

Sophia talks to Tech Inside:

Author: Anita Bakker – Principal Consultant Bumblebee Consulting