Which Automation Projects?

[Paulh_Boats]

I have started a poll for automation projects based on the low cost, open-source Arduino controller, see:

http://www.navitron.org.uk/forum/index.php/topic,5679.msg57778.html#msg57778

As it is open-source there is good potential for help and collaboration from the renewable community.

[arizmanor]

I am interested in a data logger of some sort to find out how well the solar collector is working.  I want to record various temperatures and for how long the pumps operate, and when the immersion heater in 'on'.  I have started by using "Weather Toys" as the basis for this project, but I am unable to get Netbeans to work.  I have tried asking for help on the appropriate forum, but it seems that all interest has dried up and has 'died'. 

I have voted for monitoring the mains powered devices - the two pumps and the immersion heater use.

[guydewdney]

Theres no button for 'most of the above'

TDC controller monitoring
load balancing of my inverter
export data
etc etc

[wyleu]

All of the above.

and anything that allows a system to develop to a wider community than just the control and monitoring within one dwelling.

[Paulh_Boats]

I've added "All of the above" and you can now vote for 3 items, as well as change your votes later.

Click on Remove Vote...then vote again.  If only General Elections were like this 

[briand]

Wind turbine logging includes some of those plus some extras
volts, amps and rpm of turbine
windspeed and direction - needs some processing to log average per hour say but with gustiness/variability based on readings every 10 secs for a period

temperature and pressure to help standardise results

cheers
Brian

[KenB]

Paul,

Looks like a general purpose datalogger would form the basis of most people's requirements.

There's a "datalogger" thread on the Arduino Forum, but like most threads, it lost its way (tracking Dingos in the Australian outback via GPS)

http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1196965541

Logging to SD card seems the best future-proof option.   Perhaps we should look around to see if anyone is doing a cheap SD shield as a quick way of getting started on a common platform.

LadyAda has a "WaveShield" that has an SD socket - for $22

http://www.ladyada.net/make/waveshield/index.html

One Gripe about Arduino - What bugs me is why they ever put the "bulge" on one end of what was otherwise a nice rectangular pcb.  Did they not realise how much material wastage that leads to and what it does to the manufacturing cost?


Ken

[Paulh_Boats]

Ken,

The official Ethernet shield has an SD socket, €30. The SD library is not quite ready, but if you look at the schematic the hardware is ready to roll.  That gets my vote, ethernet support might be easier also...time will tell.

Another option - update the EEPROM with kWh every 8 hours. Keeps in the 100,000 cycle lifetime limit (91 years!).


Re- the hardware footprint. If volume gets into the 100s a new PCB from the land of the rising sun will fix those problems, and it can be made to fit a case.

-Paul

[roys]

Sure most of you have seen this info, but a link to more projects using arduino kit.

http://www.instructables.com/id/Arduino/

Cheers Steve

[Paulh_Boats]

Ping...... Back to the top of the list for those who have not voted!

[wdh]

Overall, it'd be nice - considerate even, to adopt a "Keep It Simple, Stupid" approach.
While KISS is not the optimum performance approach, surely it is the best approach for:- sharing information with a group of mixed ability, code-reuse and maintainability.

Appreciating the thought, etc, already put in by those with much more relevant experience than mine...
but ...
wouldn't it make sense to think a little more modularly?
One step at a time.
And then the modules could be combined, in a pick-and-mix style, to achieve whatever specific tasks individuals might want.



Generally, I'd expect most of the datalogging interest to be of 'slow' data. People might, for example, be interested in how their hot water tank, solar collector, boiler return temperature, or whatever varied over the day, week, month or season.
I don't think there's likely to be as much interest in recording rapid "transient" behaviour, where the entire episode only lasts minutes or seconds. Controlling rapidly changing behaviour might be a popular interest, however.
Therefore as far as the "logging module" (software and hardware) goes, it shouldn't matter whether the data is coming from a temperature, voltage, sunshine or water pressure/depth sensor ... The differences there are 'just' a matter of sensors and maybe data filtering/smoothing to suit the sensor. The "back-end" is common, for sure speed is not of the essence but compared to the natural run-time of an Arduino on a 9v PP3 battery, the duration is very, very long!

I'm not sure that I properly understand the value of only accessing one's own data from the 'internet cloud' (which seems to be the pachube approach, if I understand it correctly), and/or having to write additional Java/Javascript/PHP/Ruby/Python or whatever to analyse one's own measurements does seem like, well... needing more cleverness than the job ought to require!
While I can readily appreciate the "community value" of sharing stuff like outdoor shade temperatures, sunshine hours, windspeed, etc, that data is only as good as its source. As a comparison, I can post an internet page on brain surgery, but the fact that I can post it shouldn't make it believable. Posting an outdoor temperature (to take one specific example) from an uncalibrated sensor, that is exposed to direct sun, or rain and wind, really doesn't help anyone.
But its the indirectness, the requirement to go through an intermediary (and/or to learn yet more programming languages and environments), and/or the need to run a real computer 24/7 to catch the data somehow, that makes the ethernet route seem unappealing -- at least as a starting place.
And it would also seem that a static external IP address from one's Internet Service Provider is yet another additional requirement to properly use the pachube approach.


The approach that has appealed to my simple mind is storing data in flash memory, in comma-separated-value (CSV) files (for ease of future handling), with the flash being occasionally physically transferred to a computer where the data can be easily visualised or analysed in any spreadsheet program. (Though there's no requirement for a spreadsheet -- if anyone wanted to, they could import it into MySQL, or whatever they wanted.)
Additionally, my feeling is that it would actually be simpler to store rather 'raw' data and to do any awkward scaling/linearising/conversion tasks in the spreadsheet.
Certainly, any computation with the data would be more easily done, debugged and audited on the computer. And wouldn't take up precious memory resources on the Arduino. Or obfuscate the Arduino code with all that data processing!

However, this does mean logging to flash that is formatted in a PC/Mac/Linux readable fashion.
Getting the Arduino to handle a PC filesystem seems to be a complex bit of voodoo. And I'm far from convinced that its reliable let alone easy to use ... My suspicion is that this may be the reason that the Arduino ethernet shield's SD socket isn't yet operational. Might it be that the effort and resources required for running both TCP/IP and FAT16 cripples the Arduino for any other task?

The easiest thing for users OUGHT to be to stick close to "official" Arduino developments.
But the ethernet shield's SD socket isn't (yet) supported, and as of last night none of Arduino's own three listed UK sources were even offering the official ethernet shield. (In Switzerland, it seemed to be near double the cost of an official Arduino - which seems pricey.)
Those seem to me like reasons, for now at least, NOT to restrict things to the "official" SD card !



The easiest means of handling FAT16-formatted flash would seem to be using another dedicated controller offering "dos-on-a-chip". There's an american offering claiming just that, but its expensive (compared to Arduino, anyway).

However, there does seem to be an off-the-shelf, British, packaged solution for about £15 that looks like it should be pretty easy to use.
It even uses a USB flash 'disk', eliminating various concerns - like future availability of SD (rather than SDHC) cards...
Has anyone any experience of the Vinculum Vdrive2 ? (Or its bare-board cousins?)
Info: http://www.vinculum.com/prd_vdrive1.html
Code sketch: http://www.arduino.cc/playground/Main/UsbMemory
Shop: http://apple.clickandbuild.com/cnb/shop/ftdichip?op=catalogue-products-null&prodCategoryID=54&title=VDRIVE2
And I think the trivially simple means of doing a firmware upgrade on the module demonstrates that someone has a talent for creative usability!

One small-ish but practical problem is that I know from camera experience that the most likely way of corrupting data on flash is to write to it as the camera battery is going flat...
So, one aspect of a flash-based logger might therefore be a baby ups. Or some means of monitoring its own power ...

[wyleu]

Good post wdh,

Certainly slow data is the meat and potatoes of this is sort of stuff at the moment, but I suspect it wont reamin their for long. Personally I test my event loops with musical midi data on the basis if what pops out of the far end sounds anything like the tune that went in on the input side, then the odds are the event loop is pretty good. If we can deal with that across the kind of community data rigs we seem to be constructing, then transient behaviour might become of considerable interest, who can say ? but since most of the experimentation at the moment is pretty free form it's good that it can scale a couple of decades in speed without us having to reconstruct the protocols every time we find a whole new interesting problem set to examine.

The accessing of data from the internet cloud is of tremendous advantage in that it allows standard data formats to develop quickly, we probably  don't want a situation where there are several proprietary formats arguing the toss because of their difering parentage. I watched the simple joys of the RGB world of television meet the CYMK world of printing   and the arguments  still ratttle to this day. Better we have some form of open format that everyone tries more or less to agree to so that the windy elements of the turbine world can compare and contrast data with the sun crazed advocates of the PV or the boiling hot pots of the evacuated tube or an other kind of wierd and wonderful device we haven't seen yet.
If you can do it with the cloud you'll have the clouds choice of tools available, if you do it yourself you'll have to write it all yourself. We've done the navitron system with a pachube interface cos already I don't have to write graphing code, and if I do write some I'd make it pachube compatible so it's useful to others.

The issue of calibration is certainly something that would get the scientific community shaking it's head in despair, and certainly the fact that digital solutions like one-wire mean we ain't having to calibrate op amp gain and offset tweaks on everything we do, but that is where the mass of the community will start to bear fruit. If the sensor on my solar array reads high then comparison with other systems will start to reveal it. Wether or not I cloose to do anything about it is as ever up to me but at least I know and so does everyone who might be remotely in what I have to say. How much do you think has been invested in trusted networks where such issues are important, and we get it for free cos we accept our reading might and indeed will be inaccurate. An uncalibrated sensor will still tell you if the sun is shining and to a first order that's probably good enough for most things we might want to do. The real joy is you have information from a distance and connectivity. If that connectivity dissappears, why is that? is it a failure of the sensor or is that network connection down? That's a diferent sort of data that tests components like power supply and Internet connectivity which we all assume is perfect but all know is transient. Such a community will provide some rather interesting data to beat up the ISP's with. Extend that kind of sort to the utility companies and suddenly networks of like minded users start to have tangible evidence to prsent that exists between the level of the householder and the utility. Again not bad for a dodgy solar sensor just over the hill.

I've done a fair bit of measuring over the last few years and can point at several million data points in systems of various complexity. The difficulties of trying to find out 'matching' sets of data in simple csv sets has proved to be more of a problem of library like issues rather than the often simple requirements of gathering. Did sensor 5 actually work at that stage? was it connected to the flow or return pipe? Was the real time clock unnig BST or GMT or had it reverted to Washington 1st Jn 1970 for some unaccountable reason? all of this sort of stuff meant that Itend to tag the data as early as possible with what can seem ridiculously overly involved but does at least allow reading to stand in context rather than requiring other information stored in many different places, some of which turn out to be my rather unreliable brain. It helps nd it means that the data can be pulled out of the measuring elements quickly and stored elsewhere. The method doesn't matter but it's great that I still know that it's sensor 5 in the tank on this day at that time, if I'm looking for a time when the system did something equally daft at some unknown time in the past. If the person looking for that strange kink in the system isn't me then the task becomes even harder.





You have a view and quite wonderfully I have absolutely no way of making you do anything. You do what you want to do. However if someone here gives you an idea, you cannot get rid of that, because it's the quality of the idea that appeals not the source, and you will mix it into your ideas and work accordingly. Likewise your views will influence and advantages you demonstrate will spread and we will all benefit. That's why we do it. We believe we can save many millions of kilowatts of energy simply by understanding the systems we use better and to do that we hope to construct mechanisms that allow like to be compared to like delivered by unlike.

[wdh]

Wylieu, though I applaud your wish that data should be not just shareable, but also actually shared, I just don't think that's the simplest, most basic way of doing it.

However, a little bit of 'modular programming' should cover the problem.
All one has to do is to make quite certain that the data output/display/storage (or whatever) is its own little chunk of program, with which people wanting to do something specific can 'do their own thing' and replace.

The most basic thing of all would be just to write the data to the computer using the "Serial Monitor" facility.
Doing anything different with the data shouldn't influence the program structure or construction, should it?


Incidentally, the basic and versatile method of getting data up a cable to a Mac would seem to be to use "SerialPort X" which makes any Mac serial ports (including the virtual Arduino serial port) available to Applescript for receive and transmit. And Applescript allows communication and control of (or from) almost all Mac programs...

[wyleu]

Oh don't worry It's just stuff that I'm putting a lot of thought into at the moment.

Simplicity is indeed a wonderful thing, and depending on what your up to, is the preferred Engineering solution. As has been said your perfectly at liberty to do pretty much anything you want with this stuff cos at the end of the day they're your toys, and you play as you want with whoever you want.

Modular is great, but if their is so much as a smattering of standard then it's a good idea to get to it as soon as possible and it's very easy for a quick bit of test code to morph into 'The system' or blob or any number of objects from Design Patterns or more likely Anti - Patterns, along with the debug routines doing service as interfaces. The problem tends to be that if your controlling the blocks AND the data formats the whole process can become rather feedback ridden as the change in emphasis between modules moves the responsibilities around the system. If your blocks use standards for exchange then at least you have something to allow you to break the loop and monitor and test in isolation, which shows advantages as your system grows.

Rather wonderfully we're talking about it as different approaches to knowledge gathering and the more we harvest the more we know.

Good eh?

[djh]

Linux or Mac jihad groups can port the C# to Mono if they wish. 

Sorry, although some Linux users will use mono, the jihad groups certainly won't 

It's the work of the devil, or so I've been told 

Embrace, extend and extinguish