navitron
 
Renewable Energy and Sustainability Forum
Welcome, Guest. Please login or register.

Login with username, password and session length
News: Anyone wishing to register as a new member on the forum is strongly recommended to use a "proper" email address - following recent spam/hack attempts on the forum, all security is set to "high", and "disposable" email addresses like Gmail, Yahoo and Hotmail tend to be viewed with suspicion, and the application rejected if there is any doubt whatsoever
 
Recent Articles: Navitron Partners With Solax to Help Create A More Sustainable Future | Navitron Calls for Increased Carbon Footprint Reduction In Light of Earth Overshoot Day | A plea from The David School - Issue 18
   Home   Help Search Login Register  
Pages: [1] 2 3 4 5 ... 7   Go Down
  Print  
Author Topic: TDC3-e new developments - Wookey, djh, wyleu et al. please read!!  (Read 54339 times)
Ivan
Guest
« on: July 28, 2008, 03:31:27 PM »

Wookey, djh, wyleu and others,

We've had several discussions here about the TDC3-e (which is Sorel's ethernet enabled controller). Lots of people have asked how to get the data into a data logger program etc.

Today I received the new version, but still waiting for some technical comments on what is new etc. First thing I've noticed is that Sorel have provided their own data logger software. I'll be testing this shortly, and will post some screenshots.

However, I'm keen to enable that we make the unit capable of doing what customers want. I have the attention of one of the software developers at the site, so need some input from you guys as to what I need to ask for. I'll post my current communication below. If there are any features you'd like, please specify them now! Also, please don't just tell me what you'd like to do, tell me what I need to ask for from Sorel

eg.

I'd like to be able to call the temperatures directly from a webpage....but I don't know what I need to ask for...
Logged
Ivan
Guest
« Reply #1 on: July 28, 2008, 03:34:21 PM »

FROM IVAN to Sorel:
 
> Today I received the new TDC3-e. Before I start testing it, can you
> please let me know:
 
> 1) Is the hardware/firmware different from the TDC3-e that I already have?
>
> 2)One of our customers was talking to you at the show in London. He said
> that Sorel would be offering an 'API' for this controller
> 3)Is the datalogging possible using the customer's own choice of
> datalogging program?  (Some customers are also logging sunlight
> intensity, additional temperature sensors etc)
>
> 4)Is it now possible for me to publish the S1, S2, S3 temperatures from
> my Sorel controller directly onto the Navitron website? What software
> code do I need to do this?


FROM SOREL to Ivan:

'The following information is preliminary and unverified, but I forwarded
your questions to the programer, who will answer to you directly.

1. The TDC-Ethernet's program is not different from the usual TDC. The
Ethernet function is implemented by a additional module that is plugged
into the TDC.

2. As far as I know the data stream is exported as CSV, which is
basically just numeric values without much ado, so it shouldn't be too
difficult to customize a program if necessary from external developers.

3. If your customer is a programer, he shouldn't have any problem in
using the logging function. The data is not encrypted.

4. You would need to connect the TDC via ethernet to a server and run an
application (which does not exist as of now) that uploads the received
data to some kind of Web live stream. Although this is technically not
much of a problem, this kind of application is not planned right now.

Logged
djh
Hero Member
*****
Offline Offline

Posts: 2129


« Reply #2 on: July 28, 2008, 03:56:04 PM »

Good stuff!

2. As far as I know the data stream is exported as CSV, which is
basically just numeric values without much ado, so it shouldn't be too
difficult to customize a program if necessary from external developers.

This is pretty good - it's just basic spreadsheet format for anybody who doesn't know:

1,2,3,4
5,6,7,8
and so on

It's easy to deal with. The usual mystery is what all the numbers mean, so if Sorel could provide a list that described what each number is that would be helpful.

Details of the protocol would also be useful. Does the controller just continually export this stuff, or does it need to be asked?

Quote
3. If your customer is a programer, he shouldn't have any problem in
using the logging function. The data is not encrypted.

4. You would need to connect the TDC via ethernet to a server and run an
application (which does not exist as of now) that uploads the received
data to some kind of Web live stream. Although this is technically not
much of a problem, this kind of application is not planned right now.

I can write this sort of program, as I'm sure can wookey, wyleu et al

Cheers, Dave
Logged

Cheers, Dave
Ivan
Guest
« Reply #3 on: July 28, 2008, 05:51:55 PM »

OK, I've managed to get it to connect up. Here's a screenshot of the Sorel datalogger program.

If someone can tell me how I open a CSV stream from the controller, I'll post the data. I guess it looks something like the screenshot, so I should be able to work out which number is which....



Ivan


* snapshot datalogger.jpg (38.14 KB, 707x254 - viewed 4319 times.)
Logged
Ivan
Guest
« Reply #4 on: July 28, 2008, 05:54:32 PM »

incidentally, the server built into the ethernet controller looks like this (and I can enter my adsl gateway - is this the key to publishing the data on the web?)



* tdc3e server screenshot.jpg (76.37 KB, 748x702 - viewed 1962 times.)
Logged
billt
Hero Member
*****
Offline Offline

Posts: 833


« Reply #5 on: July 28, 2008, 07:01:39 PM »

AIUI the Lantronix device is just a serial to ethernet converter. The page you're showing is just used to set the IP address etc. of the converter. Lantronix provide software which lets you set up a PC to see the ethernet device as a serial port, from that PC you could use some software to read data from the port and then publish the data from the logger.

However, it would be a lot easier to use their software, I suspect.
Logged
AndyC
Newbie
*
Offline Offline

Posts: 17



« Reply #6 on: July 28, 2008, 07:14:38 PM »

incidentally, the server built into the ethernet controller looks like this (and I can enter my adsl gateway - is this the key to publishing the data on the web?)



Sort-of!

If it's to work on the Internet rather than just your own network, it will need to know it's default gateway address, which will be your ADSL gateway. Stop me if I'm teaching granny egg-sucking, but basically how it works is that by combining its own IP address and its subnet mask, the unit can work out whether a host (i.e. other computer) it wants to communicate with is on it's own local network or on the Internet. If the host it wants to talk with is on the Internet, then the unit talks via the default gateway.

In your case the unit has an IP address of 192.168.0.50 and a subnet mask of 255.255.255.0 - from this the unit knows that it is local to any address beginning with 192.168.0.something. If you were to put your ADSL gateway's address in as the default gateway, the unit would then know that if it wanted to communicate with say Google, on 66.102.9.104, which isn't on the 192.168.0 subnet, that it needs to go via the ADSL gateway.

That's only half of it though. The trickier bit is making your unit act as a server on the Internet, rather than just on your network. The IP address range you are using (192.168.0.x) is called a 'private address range' - it's designed to work only on your network and won't be 'routable' i.e. visible on the Internet. What this means is that other people on the Internet can't type in 192.168.0.50 and get your TDC3e. Now your ADSL gateway has two IP addresses - one, which is on your local network and private, will be a 192.168.0. address. The other is on the Internet (via your ISP) and will be a public IP address, which people on the Internet can reach. What you need to do is to configure a tunnel on your ADSL gateway so that requests directed at it's public IP address are passed through the gateway to the TDC3e's private IP address. Not all gateways do tunneling, but most do - unfortunately there's no standard way of setting this up, so each brand of gateway will be different.

The other thing you'll need to worry about is that the public IP address of your gateway may well change over time - with some ISPs is stays fairly constant, with others it changes every few days. This is obviously a problem for people trying to connect. What you then need to do is either pay your ISP more for a fixed IP address (if it's an option) or do something called dynamic DNS, which is another kettle of fish!

Hope this isn't too obtuse!

AndyC.

P.S. It would be interesting to see what's under the 'server' button on the left-hand menu of the web-page you posted.
Logged
Ivan
Guest
« Reply #7 on: July 28, 2008, 10:38:39 PM »

Thanks Andy, please continue to assume that I am a grannie wanting to such eggs, and you'll pitch it correctly for my level!

Yes, I understand what you mean, and I have done the tunnelling thing, together with DynDNS.org for a webcam, so I think I could probably set that up. If anyone feels like playing around with it, I'm happy to do so, so that it is accessible from the web. I can also reinstate the webcam, which I turned off a couple of days ago, so that we can check that the IP number assigned by dynDNS is functioning correctly.

Now, can anyone tell me how I view the CSV file coming from the device. I imagine it needs to be pinged with some kind of terminal programme. Can I do it from a simple htm routine?
Logged
sleepybubble
Hero Member
*****
Offline Offline

Posts: 989


expect the unexpected, then its expected


« Reply #8 on: July 29, 2008, 12:10:55 AM »

Ivan,

Can you configure your adsl router to have a dmz or http port forward to the server on the tdc3-e, then presumably anybody could use a copy of the client software to view your data.
Will sorel give you a copy of the code for their software, which could be poked through to identify the location on the server for the csv file so that it could be interrogated directly.
Or rather will sorel give you the path to the csv file on the tdc3-e.

i.e. 192.168.0.1/data/data.csv

Writing html code to display the content of a csv file is very straight forward, but knowing the specific location of the data source is key to the issue.

Being able to interrogate the data and store it to a sturdier database than a csv file would be the ideal situation, csv files get a bit of a person to query after they pick up more than a few thausand records.
Logged

;-)
Ivan
Guest
« Reply #9 on: July 29, 2008, 12:37:41 AM »

OK, I've added the inbound service to the firewall rules. The only problem is, that there doesn't seem any obvious way to identify which incoming demands should go to the webcam (192.168.0.30) and which to the TDC3-e(192.168.0.50). Somewhere I think there should be a port allocated to each...perhaps it's allocated in the firmware of each device....Sorry, I'm slow when it comes to networking.

My current ip is 92.3.22.110.

Can anyone check to see if they have access to it?


Ivan



* saver.jpg (24.58 KB, 664x165 - viewed 1884 times.)
Logged
Ivan
Guest
« Reply #10 on: July 29, 2008, 12:40:02 AM »

P.S. I've forwarded your request to Sorel. Hopefully will have a reply tomorrow.
Logged
wookey
Hero Member
*****
Offline Offline

Posts: 3221


WWW
« Reply #11 on: July 29, 2008, 03:54:42 AM »

My request would be full protocol details. Which port to talk to it on, any commands it understands and details of the numbers it puts out.

What you've said so far suggests that all it does is write out CSV data. However for doing that over ethernet it needs to use some sort of protocol to encapsulate that data. telnet?

nmapping your IP finds two accessible ports: 80 and 9999

port 80 gives:

port 9999 gives:
MAC address 00204A92ECAE
Software version V6.1.0.0 (051122) XPTEXE
AES library version 1.8.2.1

Press Enter for Setup Mode


*** basic parameters
Hardware: Ethernet TPI
IP addr 192.168.0.50, no gateway set,netmask 255.255.255.0

*** Security
SNMP is              enabled
SNMP Community Name: public
Telnet Setup is      enabled
TFTP Download is     enabled
Port 77FEh is        enabled
Web Server is        enabled
Web Setup is         enabled
ECHO is              disabled
Encryption is        disabled
Enhanced Password is disabled
Port 77F0h is        enabled

*** Channel 1
Baudrate 115200, I/F Mode 4C, Flow 00
Port 10001
Connect Mode : C0
Send '+++' in Modem Mode enabled
Auto increment source port disabled
Remote IP Adr: --- none ---, Port 00000
Disconn Mode : 00
Flush   Mode : 44

*** Expert
TCP Keepalive    : 45s
ARP cache timeout: 600s
High CPU performance: disabled
Monitor Mode @ bootup : enabled
HTTP Port Number : 80
SMTP Port Number : 25
MTU Size: 1400
Alternate MAC: disabled

and somemore guff. Not sure if this is the router, webcam or TDC3-e, but the lantronix page talks about channel1 and the MAC adress matches, so I guess the answer is 'yes' I can talk to your TDC3-e :-)

I also appear to be able to reconfigure it from here, which may not be what you want :-)

As you say, you need to distinguish between the webcam and the TDC3-e in the router forwarding config. For now it would be easiest to disable the webcam forwarding until we have worked out (or Sorel have told you) which ports on the TDC3-e are actually available. port 9999 seems a good bet, and the config says that channel1 (the serial stream) is using port 10001, so make that accessible too. It offers to auto-increment the port (currently turned off), so we may need a range (manual says 50,000-59,999 would be used).

If I connect to port 10001 nothing happens. maybe it needs to be sent a command in order to start sending data? Clarifying this with Sorel would be very helpful.

ports 10000 and 10002 are refused a connection, so the silent, but successful connection on 10001 seems promising.

For getting data from the TDC3-e to the outside world (e.g. AMEE) there are two basic approaches. AndyC has given good details on much of this. But essentially either the TDC3 knows about the router address (which it should get by DHCP) and then makes an outward connection to somewhere on the net and sends data, or you set up forwarding as you have done and then a connection can be made from outside. The former is simpler except that the box will not get return packets telling it that packets arrived (without the router forwarding). I don't know if it can be set up to just blast UDP out into the ether without worrying about ACKs or not. If it can then that might be neatest, although you will also need some ID (the MAC?) in order to match data to a particular box/account.

The xport manual http://www.lantronix.com/pdf/XPort_UG.pdf says it can do UDP.

OK. Having read the lantronix manual (mostly) I'm reasonably sure I connected to the serial channel on port 10001 and it is set to plausible settings 115200, 8N1, no flow control, always connect. (It makes sensethat Sorel set this stuff up). So we just need to know what the serial comms protocol is then we can use that over telnet. I tried typing all keys a load of control codes and got no output. I've closed the connection so it doesn't interfere wth anyone else.

Bedtime.
Logged

Wookey
Ivan
Guest
« Reply #12 on: July 29, 2008, 11:59:32 AM »

Had a reply from Sorel. It doesn't help that I don't know anything much about programming and nor does the contact person in Sorel. However, there's been huge benefit in copying/pasting the questions raised in this thread - ie the questions that I didn't know how to ask.

Sorel have sent the protocol, and I've put it in the Navitron downloads - http://navitron.org.uk/download_images/download_66.pdf

Thanks, Wookey. Yes, it's insecure at the moment, but I'll see about securing it once we've worked out what we need to do. I think we'll need to make some changes to the Lantronix server, hence making it so you guys can access it might speed things up. I'll add the gateway settings, which hopefully make the web interface visible too.
Logged
Ivan
Guest
« Reply #13 on: July 29, 2008, 12:16:38 PM »

Here's a picture of the Lantronix server settings, as requested.


* lantronix server.jpg (39.49 KB, 743x379 - viewed 1905 times.)
Logged
AndyC
Newbie
*
Offline Offline

Posts: 17



« Reply #14 on: July 29, 2008, 12:44:41 PM »

Thanks for the picture Ivan. So it looks like you can either access the unit via telnet (old fashioned, text based) or http (web interface). I'm not sure whether this is just for administering the unit or for something more interesting like serving the data.

AndyC
Logged
Pages: [1] 2 3 4 5 ... 7   Go Up
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.20 | SMF © 2013, Simple Machines
SMFAds for Free Forums
Simple Audio Video Embedder
Valid XHTML 1.0! Valid CSS!