Wind measurement

[wyleu]

The basic difference between a spreadsheet and a database is that a database expects to store data in volumes on disks whilst a spreadsheet trys to keep it's data in programme memory. Because a spreadsheet receives it's memory allocation from the operating system it can never know if the 'memory' it's addressing is 'real' ( actual memory chips on the pc bus) or 'virtual' ( copies of blocks of memory stored on disk drive). if the spreadsheet gets large it will need more than the system memory allocated  to it so the operating system will page blocks of memory out to disk, and when it is required back it must reload that page from disk memory. Since this process is not under programme control it can't be optimized. Thus in moving from one cell to another during processing the spreadsheet might well have to reload from disk with a resulting slowing down. A database expects data in formats and orders defined by the original definitions so it can optimize the process. Indeed this is what databases are all about. The best combination is probably a spreadsheet accessing subsets of data from a database.

[charlieblair]

Thanks Wyleu. Surprisingly, that all makes perfect sense. I'll see how I get on with processing the csv file on my laptop with excel (it's actually logging onto an old desktop with no spreadsheet software). If it slows right down then I'll look into database software. Are there any good free database options available for download anywhere?

And how about whether to use 1min averages or instantaneous?

[wyleu]

Sqlite www.sqlite.org is easy and simple and
postgres http://www.postgresql.org/ is big and powerful both are free.

They both run on PC's and Linux.


As to the question of averages over instantaneous, I'm busy storing a reading every 30 seconds but I'm only doing this so we can develop a 'compression' scheme that will allow the data to be shunted around without bringing everything to a halt cos somebody asked for all the figures from all of the users cos they could.
I'm favouring a data unit ( object if you prefer) that instead of storing one value and one time and location stores a start time and value and a stop time and value. By making the start and stop times large ( an hour or two) then then results between can be inferred from the values. It makes graphs easy to plot as you just record the maxima and minima and it will compress easily from a complete data set. Therefore it would be easy to generate detailed figures for the last week and then degrade the previous data for distribution to just the max's and min's. And therby not choke a system that could be reporting from hundreds of sites.

This also indicates continual data storage over a period in that the system would not generate a data object if there was a discontinuity ( power cut or whatever) in the data. So the average solar day would probably be made up of about 10 data objects rather than the 4,000 odd I get for each sensor today.

For wind measurement you would probably be better working with average values for wind direction and rpm as they do seem to vary wildly. But once again compressing this down would still reveal trends and performance for comparative analysis without moving too many data points.