Types of Big Data
Why Big Data isn’t about financial and operational data?
- The blog intends to identify the different types of big data
- In the post we advise the separation of concerns with regard to financial and operational data.
- The material provides templates from the NIST work group for reference architecture although the templates changed after I prepared these.
Five Capability Model
Financial, Reputation, and Regulatory Threats and Weakness
The type used to run the business during transactional events used to report revenue or on expenses. In the public sector we refer to this type of data as cost and benefits.
- Record Retention, zero data loss and zero down time are all applicable for the operational scope.
- The best approach for any organization or agency is to separate concerns and DO NOT mix your operational parts of the business with your fit for purpose or analytic decision support systems.
- Choose to mix the two and you have a higher cost on every project.
- You must govern all and assume every release has a potential to influence your key controls.
- Segmenting the operational scope enables speed and agility and a smaller set of applications to invest in P1 support systems.
- Governance around the key control systems has supporting corporate policies and buy-in has greater probability.
Financial Transactions by stakeholder dependency across the life of the activities. Each output becomes the input to the next process stakeholder with a record as an output. The record retention requirements for financial transactions typically is 7 years or more.
Transaction Capabilities – International and Generic
Today, we are trying very hard to not include these points to avoid this issue with compliance or another which sends up read flags with security.
What’s my cure for these types dilemma’s? I eliminate or move the threats vectors to their rightful home. Big Data isn’t the system of record for financial management capabilities, nor the home for the party or offer management capabilities. For the same reason, a single copy of the expense and revenue transaction capability results are protected within a data store.
How does the suggestion help solve the problems with security, governance and compliance? The 5 capabilities are operational and the foundation of any companies business management system. The scope for Sarbanes Oxley and all regulatory reporting. Record retention requirements with zero data loss and zero down time are all part of the scope.
IF we scale the operational and financial information back to their system of record, integrity of the financial records and retention is not going to hinder the objectives of a big data solution. The technology and requirements are only a problem in regard to the financial and operational processing moving to a big data store.
A great example being records management outside the technology designed to retain records for an organization or agency. The management of data derived from a record is not going to attain the right retention treatment when converted back to a data expert in a data store.
Requirements Capability Template
Gary Mazzaferro supplied this template to the Reference Architecture team in NIST Big Data Work Groups in the first phase of NIST development on the subject.
A horizontal segmentation of inbound from source using 1 of 3 types of feeds over a network connection an API requires an ACL or similar connection.
The segmentation within the data mart section allows each functional group to own their connections and sharing with external parties based on the containment of the authorized users within the data mart for each function. Rather than rely on offline tickets which are merely rubber stamps without the integrity of the segregation of duties requirement and intent continues to be lost in the translation between IT and Business.
In the private sector we have an option to include the 2nd horizontal data as segmented by types and use in most organizations across the world. Most businesses are not integrating their voice, video, and data and when they are we can assume streaming (real time) feeds or unstructured in column 1 or column 3.
Column 2 Operational (see above)
Column 4 Fit for purpose anything goes information we all know and most peoples desires are in their data marts.
The requirements we hear about are best segmented by the groupings identified as the grouping will manage access and allow containment when sharing beyond the organization in a shared cloud scenario. Marketing allows other marketing stakeholders to access their information.
Business Function – Roles based access
We want to avoid situations where marketing grants access to operations data without operations having anything to do with the decision.