I have always thought that working with data irrespective of the size and algorithms used could be classified as data science. My big data module did little to help soothe my confusion. Thinking that now that I know about big data and Hadoop file systems then finally, I’m now in data science territory. To my amazement, my mentor, and chief data officer (CDO) at the bank mentioned in the group technology podcast that there are a lot of business intelligence (BI) people masquerading as data scientists. She mentioned that her team was working hard when interviewing to root out fake ones and only employ data scientists. Employing the wrong skill would cost the team a lot of time in delayed projects and unmet expectations. It was better to spend more time searching for the right candidate than rushing to placement.
Source of Information
This bothered me a lot and I finally decided to enrol for data science and machine learning fundamentals. It was an online course provided by the corporate finance institute (CFI) for a fee. I decided to start with the free version and was immediately blown away. It never occurred to me that data science was all about the creation of data-driven insights that help organisations deal with uncertainty. The type of questions it aims to answer includes what type of customers are most likely to buy? what type of market regime is being entered? How much stock should be ordered to meet the forecast next week? Or when will the company run out of warehouse stock? Looking at the above questions it is unclear why there is such a fuss between BI and data science.
BI vs Data Science
The best answer I have received thus far is that the difference is in the time frame. BI looks at past behaviour or trends. This is an equivalent of descriptive analytics where there is a need to understand what has taken place in the past. Data science, on the other hand, uses past observations (descriptive data) to make predictions, estimations, and decisions about the future. Based on this, we are likely to find BI analysts in teams that deal with reporting analytics. Data scientists are likely to be associated with value analytics and quantitative analytics where predictions need to be made within expectable risk parameters. All of this depends on the types of analysis needed from the team.
Types of Analysis
There are four types of analysis namely, (1) descriptive, (2) predictive, (3) diagnostic, and (4) prescriptive. Data scientists will exclusively work on predictive and prescriptive analysis. This is because prescriptive analysis looks at the best course of action to achieve a goal (future-focused) and predictive provides a probable state of the future or an unknown variable (future-focused). BI works with facts in data that helps the organisation know “who”, “where”, “when”, “how many”, or “what” happened in the past as shown by the data. This is called descriptive analysis. Needless to say that both data science and data analysis could be right for diagnostic analysis. This is because diagnostic analysis tells us why something is happening (the leading cause). The type of analysis depends on the skills that one has.
Domain of Knowledge
An ideal data scientist is assumed to have knowledge of (1) statistics & analysis, (2) computer science & coding, as well as (3) domain knowledge. Having two would not make you an effective data scientist at all. In fact, you would become something else entirely. Having (1) statistics & analysis and (2) domain knowledge would likely make you a data analyst than a data scientist. Likewise, having (1) domain knowledge and (2) computer science & coding would make you a software developer and not a data science candidate. Knowing (1) computer science and (2) statistics & analysis would work great in machine learning and robotic process automation (RPA). The key to knowing what type of person is likely to be a data scientist depends heavily on education & training and work experience.
Data Science Process
The distinguishing factor of data scientists is the process that they follow before providing insights. It all starts with (1) data collection and storage where they evaluate how data/information was captured to ensure data quality in the database. This is proceeded by (2) transforming data for projects where only data that relates to the project is being selected (data of interest). It is only after project data has been selected that (3) a statistical & predictive analysis could be done through building models and algorithms that spot patterns in the data. Stress testing of the model needs to be done during (4) model evaluation & data visualisation as results need to be seen in graphs or otherwise. The last step is (5) sharing insights with everyone in the form of dashboards with business users.
Conclusion
Knowing what is required would help hiring managers know exactly when to look for data analysts, software developers, and machine learning experts, instead of data scientists. The key is the type of work that is required from the team or projects. As we have seen earlier that the type of analysis plays a crucial part. This is because it is easier to get a BI analyst for both descriptive (specialisation) and diagnostic analysis (cost-effective). On the other hand, we can’t deny the need for data science for all forward-looking analyses (predictive and prescriptive). I have never interviewed a candidate for data science but have done so for data analysis. Thus, could only assume this is what the CDO was talking about when she mentioned how easy it is to differentiate between a data scientist and a BI analyst. It is all based on the process that is being followed before results/insights are published. I hope that none of the hiring managers are lazy to define the job specification properly other than just putting words to fool human resources to think you need a data scientist.
