How Big is Big Data: Part 1
Big data is another one of our glorious industry’s buzzwords and like many others it’s sadly misunderstood.
Even those who you might think would understand the term aren’t always on the right track.
Listening to Radio 5 Live yesterday evening, the interviewee was a representative of the GP community and without doubt, highly intelligent.
The topic being discussed was the recent suggestion that GPs should capture information on patient sexuality as part of routine consultations.
His position was quite clear; that this is only appropriate when it is relevant to the consultation and therefore not something that he would ask routinely during his regular wart and verruca clinic.
He made reference on a number of occasions to it all being part of ‘big data’.
Technically, that’s probably not the case as it’s more than likely the data captured would be structured; i.e. in a consistent format across all GP practices and held in a centralised system or database of some kind.
Big data is typically defined as unstructured data.
As such it can be quite challenging to compartmentalise, tag, interrogate or analyse without some clever stuff happening around it.
Unstructured data is likely to be a problem for many organisations looking to work towards GDPR compliance.
Databases with clearly defined fields don’t present the same potential headache as fileshares containing a multitude of documents with file names that don’t adhere to any specific naming convention, that could contain anything from a National Insurance number to credit card details – information that might actually have been mail-merged from a structured database in the first place.
That’s where our PII (Personal Identifiable Information) Discovery Tool comes into its own.
It reads documents sitting in fileshares, identifies PII and categorises the type of information they contain; allowing the documents to be stored accordingly, encrypted, controlled access applied, etc.
As it has done on a number of occasions, GDPR has hijacked this blog so I’ll need to return to the originally planned topic some time in the near future – and that is whether you need Big Data or Small Data to spot patterns and capture valuable insights that will help you make better decisions.
One for another day!