Friday, October 7, 2011

Working with HUGE Data Volumes

With the advent of high speed mobile broadband, comes the intricate problem of handling very high volume of data to be used for network management and optimisation. It is common to have 20 TB of network or probe recordings everyday, the question is how do you process and analyze this large volumes of data to be useful? It is a known fact that even the vendors of test and measurement equipment that supplies the systems to capture and gather this data cannot cope with the volumes in terms of processing, analysis and reporting. Now considering that you add to that all other data sources that can be used for network management and optimization such as drive testing, call records, event records, performance counters, probes, provisioning, customer support and complaints… how do you handle all this to make the most out of it?

It’s simple really, we have the answer and have proven that it is possible to handle such a high volume of data easily and very fast, it is forgetting about schema’s and tables. When we design a solution we do not think of reformatting data instead we simply keep the same formats as they were in their original form, so we end up with several data formats being loaded into our systems. Why is this effective and efficient, firstly we do not need to convert and store the data in its new format. Secondly the original format where it was created is probably the best for this data since the vendor would have spent several man years to get it right for their equipment. And lastly, we simply have clever solutions...

OK the last part was pushing it too much as a sales pitch, however it is fair to say we think differently than most companies in our field and industry and we have proven that it works well.

No comments: