navitron
 
Renewable Energy and Sustainability Forum
UK's most popular Renewable Energy Forum May 25, 2012, 09:11:50 AM *
Welcome, Guest. Please login or register.

Login with username, password and session length
News: Anyone wishing to register as a new member on the forum is strongly recommended to use a "proper" email address - following recent spam/hack attempts on the forum, all security is set to "high", and "disposable" email addresses like Gmail, Yahoo and Hotmail tend to be viewed with suspicion, and the application rejected if there is any doubt whatsoever
 
Recent Articles: UPDATE ON DECC APPLICATION FOR LEAVE TO APPEAL TO THE SUPREME COURT | Yingli Green Energy's PV Module Ranks No.2 in TUV Rheinland Energy Yield Test | Navitron Solar Showers at Glastonbury for Year 5!
   Home   Help Search Login Register  
Pages: [1]   Go Down
  Print  
Author Topic: Arduino data logging via USB  (Read 3562 times)
Paulh_Boats
Global Moderator
Hero Member
*****
Offline Offline

Posts: 2768



« on: January 05, 2009, 10:05:14 PM »

List,

Modularity: 

The Arduino has a modular hardware design, you can plug in (and stack) a variety of 3rd party hardware cards like the old PC design, except they are called shields.

The software is organised as libraries, you pick and mix libraries for a project and write some custom software to glue it together. There is an I2C library for interfacing to smart chips...its easy to talk to a real time clock for example.


One idea I have is for the Arduino to send a generic XML file to the serial port which is so easy using the Serial library built into the C language e.g. Serial.Print("Hello world");  We could hijack the Pachube XML schema or the weather station XML schemas.

You plug a laptop into the USB port, run a generic program with a command to extract the data to an XML file on disk. Then import that file into hundreds of applications that read XML. Its cheap and cheerful, minimum cost is the £12 Arduino and a USB cable...you could plumb in a fixed USB cable to a desktop PC but only read the data when the PC boots up and email it to a central server for example. Its a lot simpler than setting up a router...which quickly becomes rocket science for most people. It might not be real time, but mostly we are interested in historical data.

-Paul
« Last Edit: January 05, 2009, 10:23:18 PM by Paulh_Boats » Logged
wdh
Full Member
***
Offline Offline

Posts: 151


« Reply #1 on: January 05, 2009, 11:58:09 PM »

...
One idea I have is for the Arduino to send a generic XML file to the serial port which is so easy using the Serial library built into the C language e.g. Serial.Print("Hello world");  We could hijack the Pachube XML schema or the weather station XML schemas.

You plug a laptop into the USB port, run a generic program with a command to extract the data to an XML file on disk. Then import that file into hundreds of applications that read XML. Its cheap and cheerful, minimum cost is the £12 Arduino and a USB cable...you could plumb in a fixed USB cable to a desktop PC but only read the data when the PC boots up and email it to a central server for example. Its a lot simpler than setting up a router...which quickly becomes rocket science for most people. It might not be real time, but mostly we are interested in historical data.
Paul, sadly, I don't think that would work (at least not for Mac and Linux without a constant connection).
"Note that on Mac or Linux, the Arduino board will reset (rerun your sketch from the beginning) when you connect with the serial monitor." http://arduino.cc/en/Guide/Environment
Which I read as meaning that 'occasional' data transfer by 'occasional' serial monitor connection is a bit of a non-starter for my Mac or the old PC that continually reminds me that there are lots of pretty basic things in Ubuntu that I still haven't got to grips with - yet.
The appeal of Arduino is that its a cheap realworld/sensor/hardware "interface" for the Mac that is itself comprehensibly programmable thanks to a fairly high level language and accessible documentation.

However, the obvious limitation of Arduino is its scarcity of resources - though its kinda wonderful compared to a ZX81. In fact its more than a bit like an Acorn Atom (plus optional 6522) on a chip, and 16x faster for a fraction of the price.
But with just one k of RAM (total, for all program variables), there is a very low limit to the quantity of data one can "just hold" before offloading it, somehow.
And if data is stored simply as variables in a running program, its not possible to retrieve it "later" by serialmonitor to Mac or Linux.
However, if you stuff it into the built in EEPROM, then you do get data that survives the reset, but even that is just 500 characters-worth of unstructured data space.

Hence my readiness to shell out (a little) for the prospect of additional secure data storage. IMHO a PC/Mac/Linux readable usb flash drive seems easier than using an "unformatted" EEPROM on an I2C bus -- which seems to be the alternative. That or bog down the Arduino trying to do FAT16 DOS.
Even an ancient and economically near value-free 128Mb usb flash is on a whole different scale to the 'built-in' resources.
If only one can use it easily and without overtaxing the Arduino.

While I mentioned a CSV file for spreadsheet/database import, there's no reason one couldn't store the data in XML format in a file on the USB flash to transfer that data to computers (and thence the web).  Wink
And if you keep that data storage code together as a chunk, not mixed up with anything else, then its more straightforward to alter it to display the data, or store it in a different medium, like an SD card via the Arduino ethernet shield or via the Adafruit GPS logger shild's SD card, or that I2C connected EEPROM.
Logged
Paulh_Boats
Global Moderator
Hero Member
*****
Offline Offline

Posts: 2768



« Reply #2 on: January 06, 2009, 12:59:31 AM »

wdh,

The Arduino Environment you speak of is for programming/debugging only, with a command window at the bottom to send and collect serial data.

I will write a custom program (probably C#) to send commands and receive serial data and a simple command loop in the Arduino. Linux or Mac jihad groups can port the C# to Mono if they wish.  Wink

There is an Arduino example here:
  http://arduino.cc/en/Reference/Firmata

and a C#/Mono example here...which needs a Windows GUI interface with simple buttons to replace all the command line nonsense that nobody except Linux geeks understand (or like):
 http://www.arduino.cc/playground/Interfacing/Mono

cheers
Paul
« Last Edit: January 06, 2009, 01:03:35 AM by Paulh_Boats » Logged
hiccup
Sr. Member
****
Offline Offline

Posts: 284


« Reply #3 on: January 06, 2009, 10:13:50 AM »

Hi all.

Interesting beastie this Arduino!  Not unlike the PIC-*-WEB et al from Olimex.

My $0.02 worth is that small devices like this atre great for data capture and point control, but that data logging should be done on a real machine, either on your LAN or on the internet if you must (I'm not convinced that's a good idea though - I like to know I can get at my data years into the future!)

Manual transfer of data using SD cards or USB flash is a non starter unless it is for temporary data collection. Any manual process will soon fall into disuse.

How about using a long existing standard for this sort of thing - syslog. The server is standard on Linux etc and freeware versions for M$oft  also exist. You could even log  XML data this way.

Just a thought Smiley

Hic!
Logged

16 x Sanyo HIT250E01 into SB4000TL inverter, 2 x 20 x 58mm Navi Tubes on 22deg roof facing SSE, Gledhill Torrent RE Solar 277litre Store, TDC4 Ether Controller, Xpelair Xcell400BP HRV, Stovax Riva 66 Wood Burner
wdh
Full Member
***
Offline Offline

Posts: 151


« Reply #4 on: January 06, 2009, 10:26:15 AM »

wdh,

The Arduino Environment you speak of is for programming/debugging only, with a command window at the bottom to send and collect serial data.

...
Paul, sorry but you seem to have missed both the points I was trying to make. I'll try and be clearer!

1/ The standard Arduino data capacity is pretty tiny. 512 characters/bytes in EEPROM and whatever may be left of the 1024 characters of RAM. There simply isn't much space to store 'many' readings between computer connection events.
Therefore the standard Arduino, cannot take and retain many measurements during the period that the computer is not connected and live.
2/ From Mac or Linux, Arduino seems to perform a Reset (and therefore re-initialises all program variables), whenever it is connected to usb, or the usb connection is "woken up" on the computer coming out of sleep or reboot.  (Detail below)
Therefore the approach that you propose (of logging down a usb wire to an intermittently connected/powered computer) is inherently Windows ONLY. And therefore at odds with the open-standard cross-platform of the Arduino, let alone Wyleu's open-to-the-world ethos.


Quote
Automatic (Software) Reset

Rather then requiring a physical press of the reset button before an upload, the Arduino Duemilanove is designed in a way that allows it to be reset by software running on a connected computer. One of the hardware flow control lines (DTR) of the FT232RL is connected to the reset line of the ATmega168 via a 100 nanofarad capacitor. When this line is asserted (taken low), the reset line drops long enough to reset the chip. The Arduino software uses this capability to allow you to upload code by simply pressing the upload button in the Arduino environment. This means that the bootloader can have a shorter timeout, as the lowering of DTR can be well-coordinated with the start of the upload.

This setup has other implications. When the Duemilanove is connected to either a computer running Mac OS X or Linux, it resets each time a connection is made to it from software (via USB). For the following half-second or so, the bootloader is running on the Duemilanove. 
Extract from  http://www.nkcelectronics.com/arduino-diecimila.html
« Last Edit: January 06, 2009, 10:27:46 AM by wdh » Logged
Paulh_Boats
Global Moderator
Hero Member
*****
Offline Offline

Posts: 2768



« Reply #5 on: January 06, 2009, 01:42:39 PM »

wdh,

Ta. All is not lost though:

"The Duemilanove contains a trace that can be cut to disable the auto-reset. The pads on either side of the trace can be soldered together to re-enable it. It's labeled "RESET-EN" "

A quick snip and problem sorted! (might need to fiddle the bootloader also)


Real time controllers should be robust enough to cope with power cuts, brown outs and I might use the WatchDog timer anyway, to make sure the rainwater pump does not get stuck in 2nd gear.

Memory: there is plenty of space for meter readings in the EEPROM (gas/elec/water) and a circular buffer could keep just the last 20 readings (once every half hour say).  As Ken said elsewhere part of the fun is squeezing a lot out of a cheap package.

BTW I think you will find Neil Armstrong landed the Eagle on the moon with just 2K words of RAM in the flight computer.
Logged
wdh
Full Member
***
Offline Offline

Posts: 151


« Reply #6 on: January 06, 2009, 03:04:47 PM »

... All is not lost though:

"The Duemilanove contains a trace that can be cut to disable the auto-reset. The pads on either side of the trace can be soldered together to re-enable it. It's labeled "RESET-EN" "

A quick snip and problem sorted! (might need to fiddle the bootloader also)

Thanks, I hadn't spotted that addition to the 2009 ("Duemilanove") version specification.  Roll Eyes
However, I had thought most of the interest here revolved around the current (1.16) Freeduino which is I believe a clone of the year 2000 celebrating-10,000-Arduinos-built ("Diecimila") version that does have the auto-reset, but lacks that bow-tie (or however the cut/resolder trace is implimented). 
http://arduino.cc/en/Main/Boards
EDITed to correct my rotten Italian !  Embarrassed


Quote
Real time controllers should be robust enough to cope with power cuts, brown outs and I might use the WatchDog timer anyway, to make sure the rainwater pump does not get stuck in 2nd gear.
Does a watchdog timer mean using Ladyada's bootloader rather than the standard one?

Quote
Memory: there is plenty of space for meter readings in the EEPROM (gas/elec/water) and a circular buffer could keep just the last 20 readings (once every half hour say).  As Ken said elsewhere part of the fun is squeezing a lot out of a cheap package.

BTW I think you will find Neil Armstrong landed the Eagle on the moon with just 2K words of RAM in the flight computer.
Its wonderous indeed what can be achieved with the Arduino resources.
Its just that when it comes to datalogging, the space for data (without augmentation) is a bit on the skimpy side.
Just as the LEM computer wasn't quite able to handle the rendezvous radar being used (Aldrin's self-preservation precaution) at the same time as the descent radar, so one has to recognise the unit's limitations, and work with them, around them, or change them!
« Last Edit: January 08, 2009, 11:40:22 AM by wdh » Logged
Paulh_Boats
Global Moderator
Hero Member
*****
Offline Offline

Posts: 2768



« Reply #7 on: January 06, 2009, 09:51:36 PM »

wdh,

I've carefully traced the PCB on my Nuelectronics Freeduino 1.16 board and the CRS capacitor is connected to the Reset line (CRS - reset capacitor).

Whilst there is no easy way to cut the track, the programming notes say a communicating program on a PC just has to open the serial port (causing a reset) then wait one second. During that second the boot loader sees there is no programming data and it jumps to the loaded program. Then the PC program can send/receive data.

Another option is the Ladyada bootloader which fixes the reset issues.

-Paul
Logged
wdh
Full Member
***
Offline Offline

Posts: 151


« Reply #8 on: January 07, 2009, 10:55:37 AM »

... the programming notes say a communicating program on a PC just has to open the serial port (causing a reset) then wait one second. During that second the boot loader sees there is no programming data and it jumps to the loaded program. Then the PC program can send/receive data. ...
Paul, at the risk of flogging a dead horse, when you say that on (usb) reset "it jumps to the loaded program", I have understood this to mean that it restarts the previously loaded program (thereby initialising variables, and trashing all data being held in simple program variables) - just like pressing the reset button! 
Have you found something that specifically says that it resumes the loaded program, with program variables intact?
Logged
Paulh_Boats
Global Moderator
Hero Member
*****
Offline Offline

Posts: 2768



« Reply #9 on: January 07, 2009, 11:43:40 AM »

... the programming notes say a communicating program on a PC just has to open the serial port (causing a reset) then wait one second. During that second the boot loader sees there is no programming data and it jumps to the loaded program. Then the PC program can send/receive data. ...
Paul, at the risk of flogging a dead horse, when you say that on (usb) reset "it jumps to the loaded program", I have understood this to mean that it restarts the previously loaded program (thereby initialising variables, and trashing all data being held in simple program variables) - just like pressing the reset button! 
Have you found something that specifically says that it resumes the loaded program, with program variables intact?

I'm not sure yet - I had a quick look at the Ladyada bootloader code and it can distinguish between a power reset, watchdog reset, or toggling of the Reset pin so maybe there is a smart boot up that retains program data.

Personally I think the auto-reset is a painful hack and I would disable it by cutting the track. Its unreliable as there are reports of comms programs having triggered the bootloader accidentally, and the Ladyada bootloader fixes that a bit. Pressing the reset button to enable the bootloader is secure and reliable.
Logged
Pages: [1]   Go Up
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.16 | SMF © 2011, Simple Machines Valid XHTML 1.0! Valid CSS!