Explosion of data envelops man in the street
IT is hardly news that our growing dependence on technology has resulted in a dramatic explosion in the volume of data we create, use and share. Of course, we also have to manage, organise, retain, protect and secure that same data from theft and privacy intrusion, and to comply with the regulatory environment, business and accountability requirements.
More interestingly, the data explosion has had an impact on areas outside traditional ICT circles, effectively turning data management and compliance into an issue for the man on the street.
According to IDC, the world's volume of data doubles every 18 months and will reach 18,000 exabytes (one million terabytes) by next year. A single exabyte is estimated to be equal to the information contained in 12 stacks of books extending from Earth to the sun.
This growth is being driven by new classes of digital information that require more sophisticated storage requirements in areas including e-health, (interoperability, electronic storage/retrieval of records), e-education and digital entertainment.
Organisations have been seeking effective and efficient ways to handle their data for years. Now they must also contend with the cost of complying with the regulatory environment, including laws relating specifically to records, archives, access, privacy, evidence, electronic commerce, data destruction and protection, to name a few.
It has been asserted there are more than 450 acts of parliament in Australia that contain provisions dealing with the retention of records. As a result of the global financial crisis, tighter fiscal policies are likely to drive requirements for better data management and retention to mitigate future instances of financial mismanagement.
The cost of everyday data storage has fallen dramatically in recent years, with a 2000GB drive today being around the same size, weight and cost of an 18GB drive in 1998. Or consider that a single megabyte, which was worth $10,000 in 1956 and fell to around $300 in 1983, sells for just $0.0005 today, when a 250GB drive retails for around $125.
But there are other "hidden" costs of storage: the physical space, power consumption and environmental impact of all this data, not to mention the human effort in gathering, analysing and managing so much information.
The ACS supports the development of green data centres that use technology advances to minimise the carbon footprint.
Facebook has announced its first data centre will be among the world's greenest when it opens in Oregon. The new centre will have a Power Usage Effectiveness rating of just 1.15, which means that for every 1.15 watts that go through the meter, 1.00 watt is delivered to the computer infrastructure.
The ACS believes Australia should consider the introduction of PUE guidelines. The Australian government could lead by adopting this system as a standard benchmark within government data centres and by encouraging its use within private enterprise.
Another implication of the growing data challenge is that organisations will increasingly outsource their data storage. They need to.
If Australian corporations choose to reduce costs by storing data offshore, they must take responsibility for addressing the issues of reliability, access, security and privacy.
ICT professionals responsible for operating and managing data resources need to apply the latest technologies and strategies to optimise efficiency and integrity. This will include server virtualisation, enhanced caching and disk de-duplication as part of back-up and mirroring facilities, and analytical and reporting tools with high levels of automation.
We are only beginning to understand the implications of the data explosion. We need to continue this process, work with government to address the impact around power efficiency, privacy and security issues, as well as the blurring of the lines between data devices for work and home.
With exponential growth in data volumes a certainty, ICT professionals have a responsibility to deliver systems and processes that support users in effectively managing and accessing information without being overwhelmed.
Our ability to deliver on this charter will have a significant impact on productivity and, ultimately, our quality of life.
Anthony Wong
The Australian, February 9, 2010