navitron
 
Renewable Energy and Sustainability Forum
Welcome, Guest. Please login or register.

Login with username, password and session length
News: Anyone wishing to register as a new member on the forum is strongly recommended to use a "proper" email address. Following continuous spam/hack attempts on the forum, "disposable" email addresses like Gmail, Yahoo and Hotmail tend to be viewed with suspicion, and the application rejected if there is any doubt whatsoever
 
Recent Articles: Navitron Partners With Solax to Help Create A More Sustainable Future | Navitron Calls for Increased Carbon Footprint Reduction In Light of Earth Overshoot Day | A plea from The David School - Issue 18
   Home   Help Search Login Register  
Pages: [1] 2 3 4 5 6   Go Down
  Print  
Author Topic: RaspberryPi Python3 App for uploading Eco-Eye SmartPV serial o/p to PVOutput.org  (Read 48214 times)
JST200
Jr. Member
**
Offline Offline

Posts: 58


« on: February 28, 2014, 11:06:54 AM »

          I am changing this first post to provide me a mechanism to provide new releases. The original contents of this pos is below the release documentation.

        RELEASE DOCUMENTATION

        V6-00-02

        This is a major release.  It is required if you wish to use the weather option using OpenWeatherMap, as WeatherUnderground has removed its free access.
        It also supports the use of SBFspot (which is an open source project to get actual and archive data out of an SMAŽ inverter over Bluetooth).
        It also provides various bug fixes.

        To use OpenWeatherMap will require you to get a new API from the OpenWeatherMap webiste (http://openweathermap.org/). To do so:
        • click on the "Sign up" option to create an account.
        • After your account is created you should receive an email which will contain your API key.
        • If not, then just "Sign in" to you new account and, on your home page, and click "API keys" (note: this is about a 3rd of the way down the page). The page that opens will contain your key.

        To use the new Weather service you will need to configure eespvo with the new key:
        • Using eespvo's web interface, open "eespvo > Settings > optional" and select "Weather settings".
        • Enter your new key in the "API Key" box. You can use cut and paste from the OpenWeatherMap API keys page to this field
        • You will also need to specify your location. To do this you can try just entering your location and OpenWeatherMap should try and find it's nearest weather station to you. However, I would suggest going to the OpenWeatherMap webpage and selecting "Maps > Weather maps", zoom the map in to your location and then select the nearest station to you location. Use this station as your eespvo location.

        To use the SBFspot service follow the instructions here: http://github.com/SBFspot/SBFspot/wiki, you can get the kit from here: http://github.com/SBFspot/SBFspot. Just google "SBFspot" for various other sources of info.
        Once you have it configured an running on your Pi, then you can enable it's use opening "eespvo > Settings > optional", selecting "SBFspot settings" and selecting "Enable support". Hover over, or click on. the various options to see their function.

        Installation

        To install this version I recommend the following method:

        • copy the new files from dropbox (https://www.dropbox.com/sh/8du4mqpqkrpdw3k/AACBkZVefE3hKMPoHG0abbr_a?dl=0)
        • if you are a an existing user then:
          • just copy the contents of "home-pi-eespvo" over the files already in the folder"/home/pi/eespvo"
          • If you are going to use the new Weather service then, using the eespvo web interface, select "eespvo > settings > main" and deselect "Allow eespvo to start". This will allow you to change the Weather settings (as described above) before eespvo starts running. This will avoid eespvo Weather related error messages.
          • reboot the Pi
          • if using the new Weather service, then configre it's settings once the Pi is up and running. Once you have done so, you can now re-select "Allow eespvo to run"
        • if you are a new user, then follow the these instructions:
          • the files in "home-pi-eespvo" go in the folder"/home/pi/eespvo"
          • the files in "var-www-html" go in the root folder "/var/www/html".
          • the file in "etc" goes in the root folder "/etc". This is the file that runs eespvo on boot. However, if you use a different method to do so, then skip this step.
          • Now reboot the Raspberry Pi. After booting eespvo will create a initial version of "/work/settings.json" and then wait to be told to start.
          • eespvo can be told to start through the web interface using the "Settings > Main" form, selecting the "Allow eespvo to start" checkbox and then selecting "Submit". Note: Connecting to the web interface is password protected. The initial password is "eespvo".
          • However, before allowing eespvo to run, you may want to configure it's options. These are set using the "Settings > Optional" form

        Note: to help you move files into the "root" folders, don't forget the trick of running "lxterminal" and using the command "sudo pcmanfm". This will run a privileged version of Folder Manager which has the necessary rights to copy files into "/etc/" and /var/www/html".

        Changes

        Here is a full list of changes:

        • Features:
          • Change from using Wunderground to OpenWeatherMap
          • Add support for SBFspot - Bluetooth reader of Power and Volts from SMA Inverters
          • Add variable for max. no. of data points in web graphs
          • Add debug feature write out Wunderground JSON data
          • Add debug feature write out Nest JSON data
          • Optimise Web Graph presentation
        • Bug Fixes
          • Ensure Immersion doesn't calc power when Gen power is zero
          • Fix "Bad request 400: Missing required energy generation field" for first Add Output of day
          • Fix missing Weather variables wuCity, wuWeaAct and wuWeaHld
          • Change network timeout default to 45sec to improve PVOutput and Weather timeout issues
          • Change web interface default 4sec history count to 900 to give max 2 hour view
          • Change web interface default 5min history count to 865 to give max 4 day view
          • Conform shared memory folder name variable to standard
          • Remove unnecessary ":" from email "Send To" formatting (i.e. change from "To:" or "To")

        V4-05-17

        Please note: This is a major release, with significant changes to it's operation.

        This release completely changes the way that eespvo is configured. It now uses a web interface to set all the operational settings. As part of this implementation eespvo no longer uses the setup file "eespvo.ini", rather it uses a file called "settings.json", which is modified via the web interface.

        I must start by saying a big thank-you to CLIFFMCL who has helped tremendously with this version. He has suggested significant and valuable changes and bravely tested each interim version as I released it (sometimes twice in a day!!).    genuflect

        New Features

        The major features of this version are:

        • Web Interface - the use of a web interface to control and configure eespvo
        • Jessie - built to run on the latest Jessie based Raspian image (I think this version of eespvo is backwards compatible with the previous version of Raspian, but not certain)
        • Shared memory - the use of shared memory for the transfer of real time data from eespvo to the web interface. This removes the writes to the SD Card every 4 secs, for the web interface's realtime data display, and should help extend the SD Card's life
        • PVOutput.org - the removal of the requirement to use the pvoutput.org website, allowing the use of just the web interface to show eespvo output
        • Energy used - add realtime daily energy use to the web interface
        • External IP Address - eespvo will identify the external address as presented t the web and inform you if it changes. This is useful if you want to access eespvo across the internet
        • Nest Thermometer - add Nest Thermometer support. This provides a separate Nest specific Dashboard  that shows: Target temperature, Current Temperature, Outside Temperature, Humidity and Boiler State. If you have PVOutput donator status then Target Temperature, Current Temperature, Central Heating Energy used and Hot Water Energy used are added to the pvoutput.org Extended Data graph
        • PVOutput donation status - automated detection of your PVOutput donation status. This automatically enables PVOutput's extended features (see http://www.pvoutput.org/help.html#donations ). Of specific interest are the increase in the number up uploads that can be made at one time and the addition of extended data support. The increase in the number of uploads means eespvo can sustain a network, or pvoutput.org, outage for approximately 8 hours, without losing data. This is limited to 2.5 hours without donation status
        • Realtime Volts - the ability to update the Voltage value in realtime. Currently this is read from a user supplied, JSON formatted, file
        • Single logfile - change of the log file to a single file
        • plus a significant number of other internal changes and fixes

        Installation

        To install this version I recommend the following method:

        • if you are already running a previous version of eespvo, rename your existing "eespvo" folder to something else (e.g. "eespvo org". This ensures that should the installation go wrong you can just rename the "eespvo org" back to "eespvo")
        • copy the new files from dropbox (https://www.dropbox.com/sh/8du4mqpqkrpdw3k/AACBkZVefE3hKMPoHG0abbr_a?dl=0) as follows:
          • the files in "home-pi-eespvo" go in the folder"/home/pi/eespvo" as previously
          • the files in "var-www-html" go in the root folder "/var/www/html". This is different from before, as the implementation of Apache, under Jessie, has moved the default location for it's files to the sub-folder "/html".
          • the file in "etc" goes in the root folder "/etc" as before. This is the file that runs eespvo on boot. However, if you use a different method to do so, then skip this step.
          Note: to help you move files into the "root" folders, don't forget the trick of running "lxterminal" and using the command "sudo pcmanfm". This will run a privileged version of Folder Manager which has the necessary rights to copy files into "/etc/" and /var/www/html".
        • Now reboot the Raspberry Pi. After booting eespvo will create a initial version of "/work/settings.json" and then wait to be told to start.
        • eespvo can be told to start through the web interface using the "Settings > Main" form, selecting the "Allow eespvo to start" checkbox and then selecting "Submit". Note: Connecting to the web interface is password protected. The initial password is "eespvo".
        • However, before allowing eespvo to run, you may want to configure it's options. These are set using the "Settings > Optional" form


        V4-05-19

        This is a minor release that only affects anyone who uses the Nest support.

        It fixes a problem where the Nest process would appear to stall.
        It also limits the Nest Web Interface temperature displays to a single decimal point.

        Installation

        To install this version I recommend the following method:

        • copy the new files from dropbox (https://www.dropbox.com/sh/8du4mqpqkrpdw3k/AACBkZVefE3hKMPoHG0abbr_a?dl=0)
        • if you are a an existing user then just copy the contents of "home-pi-eespvo" over the files already in the folder"/home/pi/eespvo" and reboot the Pi
        • if you are a new user, then follow the these instructions:
          • the files in "home-pi-eespvo" go in the folder"/home/pi/eespvo"
          • the files in "var-www-html" go in the root folder "/var/www/html".
          • the file in "etc" goes in the root folder "/etc". This is the file that runs eespvo on boot. However, if you use a different method to do so, then skip this step.
          • Now reboot the Raspberry Pi. After booting eespvo will create a initial version of "/work/settings.json" and then wait to be told to start.
          • eespvo can be told to start through the web interface using the "Settings > Main" form, selecting the "Allow eespvo to start" checkbox and then selecting "Submit". Note: Connecting to the web interface is password protected. The initial password is "eespvo".
          • However, before allowing eespvo to run, you may want to configure it's options. These are set using the "Settings > Optional" form

        Note: to help you move files into the "root" folders, don't forget the trick of running "lxterminal" and using the command "sudo pcmanfm". This will run a privileged version of Folder Manager which has the necessary rights to copy files into "/etc/" and /var/www/html".

        V4-05-17

        Please note: This is a major release, with significant changes to it's operation.

        This release completely changes the way that eespvo is configured. It now uses a web interface to set all the operational settings. As part of this implementation eespvo no longer uses the setup file "eespvo.ini", rather it uses a file called "settings.json", which is modified via the web interface.

        I must start by saying a big thank-you to CLIFFMCL who has helped tremendously with this version. He has suggested significant and valuable changes and bravely tested each interim version as I released it (sometimes twice in a day!!).    genuflect

        New Features

        The major features of this version are:

        • Web Interface - the use of a web interface to control and configure eespvo
        • Jessie - built to run on the latest Jessie based Raspian image (I think this version of eespvo is backwards compatible with the previous version of Raspian, but not certain)
        • Shared memory - the use of shared memory for the transfer of real time data from eespvo to the web interface. This removes the writes to the SD Card every 4 secs, for the web interface's realtime data display, and should help extend the SD Card's life
        • PVOutput.org - the removal of the requirement to use the pvoutput.org website, allowing the use of just the web interface to show eespvo output
        • Energy used - add realtime daily energy use to the web interface
        • External IP Address - eespvo will identify the external address as presented t the web and inform you if it changes. This is useful if you want to access eespvo across the internet
        • Nest Thermometer - add Nest Thermometer support. This provides a separate Nest specific Dashboard  that shows: Target temperature, Current Temperature, Outside Temperature, Humidity and Boiler State. If you have PVOutput donator status then Target Temperature, Current Temperature, Central Heating Energy used and Hot Water Energy used are added to the pvoutput.org Extended Data graph
        • PVOutput donation status - automated detection of your PVOutput donation status. This automatically enables PVOutput's extended features (see http://www.pvoutput.org/help.html#donations ). Of specific interest are the increase in the number up uploads that can be made at one time and the addition of extended data support. The increase in the number of uploads means eespvo can sustain a network, or pvoutput.org, outage for approximately 8 hours, without losing data. This is limited to 2.5 hours without donation status
        • Realtime Volts - the ability to update the Voltage value in realtime. Currently this is read from a user supplied, JSON formatted, file
        • Single logfile - change of the log file to a single file
        • plus a significant number of other internal changes and fixes

        Installation

        To install this version I recommend the following method:

        • if you are already running a previous version of eespvo, rename your existing "eespvo" folder to something else (e.g. "eespvo org". This ensures that should the installation go wrong you can just rename the "eespvo org" back to "eespvo")
        • copy the new files from dropbox (https://www.dropbox.com/sh/8du4mqpqkrpdw3k/AACBkZVefE3hKMPoHG0abbr_a?dl=0) as follows:
          • the files in "home-pi-eespvo" go in the folder"/home/pi/eespvo" as previously
          • the files in "var-www-html" go in the root folder "/var/www/html". This is different from before, as the implementation of Apache, under Jessie, has moved the default location for it's files to the sub-folder "/html".
          • the file in "etc" goes in the root folder "/etc" as before. This is the file that runs eespvo on boot. However, if you use a different method to do so, then skip this step.
          Note: to help you move files into the "root" folders, don't forget the trick of running "lxterminal" and using the command "sudo pcmanfm". This will run a privileged version of Folder Manager which has the necessary rights to copy files into "/etc/" and /var/www/html".
        • Now reboot the Raspberry Pi. After booting eespvo will create a initial version of "/work/settings.json" and then wait to be told to start.
        • eespvo can be told to start through the web interface using the "Settings > Main" form, selecting the "Allow eespvo to start" checkbox and then selecting "Submit". Note: Connecting to the web interface is password protected. The initial password is "eespvo".
        • However, before allowing eespvo to run, you may want to configure it's options. These are set using the "Settings > Optional" form

        Settings

        It is recommended that you change the password to something other than eespvo - especially if you plan to use eespvo over the web. This is done through the "Settings > Main" form, selecting "Web Interface Control Settings" (which will show the available settings), typing your new password into the "Password" field, then selecting "Submit". If the "Password" field is cleared (i.e. empty), then password verification is skipped.

        When using the web "Settings > Optional" form, each topic can be selected by clicking on the option. At which point all the relevant settings will be shown. The first "Optional" setting, for each option, is the "Enabled" setting, which "turns on" that particular feature. If an option is enabled, then the top level list will show this.

        There are three levels of settings: Standard, Advanced and Expert. Which level is displayed is selected on the "Settings > Main" form, selecting "web interface control settings" and then selecting the required "Settings Level". Most users should only need to use the "Standard" settings view. Use settings visible via the "Advanced" view with caution. Settings visible via the "Expert" view are really intended for debugging purposes and should only be used with extreme caution, as the effects may not necessarily be what you expect.

        As you select each setting a popup will provide additional information about that setting. Hopefully this makes each setting self explanatory.  fingers crossed!

        Not all settings in the file "settings.json" can currently be set via the web interface. However, few of these would ever be set by anybody other than me during testing! But I will add them to the web interface over time.

        Note:
        It is strongly recommended that the web interface is used to make changes to the settings
        However, it is still possible to make changes by editing the "settings.json" file. This file is located in the folder "/home/pi/eespvo/work". Both the folder and file are created on first run of eespvo.

        Be warned: the JSON formatting is extremely prescriptive. The slightest format error will stop the file being read. Also, there are no comments or other indications in the file to describe the effect of any of the settings. Therefore manual changes should be done with extreme caution.

        Realtime Voltage
        Realtime voltage data can be used by eespvo. This is achieved by reading a user supplied file.
        The file must be created in the "/eespvo/sharedmem" folder (note: this folder does not exist until eespvo has been run once.)
        The file must use JSON format (it's contents must look like this: {"volts": 240.0} )
        Real time volts is selected using the "Settings > Optional" page, selecting "Volts Settings", selecting "Enable realtime support" and then selecting the "Volts realtime input > File" checkbox .

        External IP Address
        CLIFFMCL has already noticed that the current mechanism, used to get the external IP address, may return an IPV6 address (thanks Cliff). If this happens, then modify the url settings eespvo uses as follows:
        • turn on the "Expert" view, using "Settings > Main > Web Interface Control Settings > Settings Level > Expert"
        • change the url used to http://ipv4.myexternalip.com/json, using "Settings > Optional > IP Settings > Expert Settings > URL used to get External IP Address"

        The web Interface.
        Web interface examples:

        The eespvo Dashboard:


        The eespvo "Settings > Main" page:


        The eespvo "Settings > Optional" page:


        The PVOutput Dashboard (Live Production):


        The PVOutput Dashboard (Extended Data):


        The Nest Dashboard:


        Good luck!
        Cheers, Jim


        V3-08-01

        Major features are:
        • Enable location of webserver folder to be set in .ini file.
          If you are running eespvo on something other than a Raspberry Pi, this allows the webserver folder to be located somewhere else.
          It is recommended that Pi users don't change this location.
          (Related switch is: wwwFileDir - www directory (def: /var/www/) )
        • Import GPIO module as and when requested on the fly.
          If you are running eespvo on something other than a Raspberry Pi it may not have the correct import modules required for flashing the Activity LED. This change ensures that the import only occurs when the FlashLED option is selected.
        • Catch unexpected data in USB data streams.
          Record, but don't crash (not implemented for Immersion USB stream yet).
          [Related switches are: usbErrLmt (Integer): - USB Consecutive Input Error Limit (def:50)
                                               dbgUSBRanChar  (Boolean):  - Randomly insert char into USB Data Stream (def:False) - for debug purposes
        • Tidy up constants setup code
          My interest only - though it might help others set variable in the code.

        The new version is available via dropbox here: https://www.dropbox.com/sh/8du4mqpqkrpdw3k/AACBkZVefE3hKMPoHG0abbr_a?dl=0

        As usual, copy the "eespvo v3-08-01.py" into the eespvo folder. I suggest you then rename the existing "eespvo.py" to "eespvo.py old" (so you can easily revert back to it if something doesn't work). Rename "eespvo v3-08-01.py" to "eespvo.py" and reboot. I have left "eespvo v3-07-04.py" on the share too, just in case.



        V3-07-04

        Major features are:
        • provides a web based graphical interface of real-time data showing current Generation, Consumption and Imported/Exported Watts. It also shows a historical graph of the real-time data. It works on PCs, tablets and phones. The only note of caution is that it doesn't auto-update every 4secs when using Microsoft's Internet Explorer. All other web browsers seem to work fine. This interface is the 1st element of providing a web based interface for configuration.




        To use the web interface you need to do the following:
        • install the Apache Webserver, with php support, on your Pi (tip: in lxTerminal use the commands "sudo apt-get install apache2 php5 libapache2-mod-php5", "sudo service apache2 restart")
        • update eespvo to version v3-07-04
        • add the folder "www", and it's contents, to the "eespvo" folder.
        • copy the file "index.php" and add the folders "components" and "pages", into the root folder "/var/www" (tip: to copy files into the root folder "/var/www", in lxTerminal use the command "sudo pcmanfm&". This runs the File Manager in privilege mode and allows access)
        • Use the IP address of your PI from your browser to access the webpage (tip: PM me to get the password to access the web page

        Announcement: I can now provide a full Raspian image with eespvo preloaded. The image fits on an 8GB SD card and supports both the Raspberry Pi B and the new Raspberry Pi 2. It has Apache Webserver (with php support), VNCServer (for remote terminal access) and SAMBA (for Windows File Access) already installed and configured. Drop me a PM if you are interested.

        Note: I have removed all the distribution files as attachments to this post. It was getting too complicated to keep all the required files up to date. Therefore I have made them all available via dropbox: https://www.dropbox.com/sh/8du4mqpqkrpdw3k/AACBkZVefE3hKMPoHG0abbr_a?dl=0

        V3-06-06

        Apologies for the extreme delay in distributing this new release!

        To be honest, I didn't think anyone was particularly interested in the app, so although I have been working on features for myself I haven't been uploading them.

        Anyway, here is the latest release V3-06-06. There have been some quite significant additions to the settings and functionality available. All the settings that can be configured are documented here: http://www.navitron.org.uk/forum/index.php/topic,22190.msg286557.html#msg286557

        Major features are:
        • works with the latest release of Raspian (which works with both the Pi Model B and Pi 2)
        • allows the 4sec data to be logged to daily files (the number of files produced can be controlled)
        • allows the 5min data to be logged to daily files (the number of files produced can be controlled)

        Major fixes are:
        • a bug which would cause eespvo to crash once a week or so, with "Bad Status Line" has been fixed.

        Notes:
        • It appears that there are slight differences in the way the NOOB and RASPIAN implementations of the OS work. I know of one installation that can't get the eespvo.scr to work. The only difference we can see is that their OS was created using NOOB and I use RASPIAN. I would recommend using RASPIAN if you can, as once you have the hang of NOOB it's quite easy to use and it seems to provide a much simpler update and upgrade capability than NOO (where you have to muck abou unloading .zip files)
        • I have added an example of the .ini file I use (eespvo.ini.example.txt) in case it helps anyone
        • If anyone would rather pick up the files from dropbox, rather than use the attachments below, then they can do so from here: https://www.dropbox.com/sh/okuxk5eoz1y95as/AAAEL7Fn0o7e1CFOUfmcYXkea?dl=0
        • Because of fixing the crash problem the use of eespvo.scr, with its auto restart capability, has become less important.  The app is (or should be!) very resilient to problems. Therefore an alternative way to start the app is to use CRON.

        V3-03-07

        Made less sensitive to eespvo.ini format.

        V3-03-05

        First release.

                                                                                                                                                                                         

        Original post contents

        This thread has been moved from:

        http://www.navitron.org.uk/forum/index.php/topic,21807.msg257620.html#msg257620




        Hi,

        let me start by saying a massive thank you to IanH64 for his initial code. It kick started me into writing my own version which I offer as a possible alternative to Ian's.

        I accept that it is by no means the greatest python3 code - so it isn't very object oriented and probably doesn't use python best coding practices. However, it works (for me at least) and has been running in one form or another since early Jan. It hopefully uses snippets of code that might help you with other projects, e.g. how to send smtp emails and blinking the ACT Led.

        It's features are:
        • it is designed to run completely headless, though you can run it interactively if you wish

        • it will survive network outages, storing the intervening data, for up to 150mins or,if you donate to pvoutput.org, 500mins.
        • it can be configured to flash the Pi's ACT LED to show its status
        • parameters can be configured via a .ini file or within the program itself if you prefer. The .ini file being the recommended way.
        • it can be configured to get weather data from local (to you) weather stations from wunderground.com, this allows it to get not only temperature readings (ala pvoutput.com's native solution) but also the prevailing weather conditions at a given time of day set by you
        • it can be configured to report encountered errors to you via smtp mail
        • it will survive and report all errors, even programming errors, in a log file
        • it has 6 levels of logging, should you wish to see more of what it is doing
        • I have commented it fairly heavily so that should you wish to tinker, then you have a chance of understanding what it is trying to do
        One note about the Blinking LED feature. The LED is shared with the file systems access to the SD Card. Therefore the blinking may sometimes be a bit inconsistent.


        Here are the instructions of how to set it up. Please note I pretty consistently use the desktop interface, so much of the instructions will be based around this. Also, there are loads of sites that explain how to do some of the detail things, so I will not cover them in too much detail.

        So to set it up you have to do the following:

        • Install the python3 serial interface libraries.
          To do so, run LXTerminal and type "sudo apt-get install python3-serial" (without the quotes!).
          PLEASE NOTE! Be absolutely sure you type "python3-serial" and NOT "python-serial". If you get it wrong it can be quite a pain to correct (guess how I know!)
        • Create the folder "/home/pi/eespvo".
          To do so: run File Manager > click on the folder "pi" > select "File > Create New > Folder". In the pop-up box enter "eespvo"
        • Copy the attached file "eespvo vn-nn-nn_py.txt" below into the new folder "/home/pi/eespvo" and rename it to "eespvo.py".
        • Copy the attached file "eespvo_scr.txt" below into the new folder "/home/pi/eespvo" and rename it to "eespvo.scr".
        • Make eespvo.scr an executable file.
          To do so: run File Manager > right click on eespvo.scr and select Properties. Select the Permissions tab and tick the "make the file executable" box.
        • Copy the attached file "eespvo_ini.txt" below into the new folder "/home/pi/eespvo" and rename it to "eespvo.ini".
          This file will then need editing to meet your own requirements.
          To do so: run File Manager > right click on eespvo.ini and select LeafPad
        • Edit the file "/etc/rc.local" to add a line that will auto-start eespvo.py at boot.
          To do so: run LXTerminal. Type "sudo nano /etc/rc.local". This will run the nano editor. Insert the following line before the line "exit 0" line: "(sleep 30; sudo /home/pi/eespvo/eespvo.scr)&"  - without the quotes!

        • reboot the pi.
          If you have selected the LED flash, then it should start to flash after about 1min after the reboot. If you have selected an email you should receive an email a couple of mins after the reboot.


        Hopefully this the above will be enough to get you going. As noted in the code, this code is supplied without warranty for being fit for purpose under the GNU GPL v3 license or later.  I will answer any questions as and when I can - though of course no guarantees!

        Good luck!
        Cheers, Jim
        [/list][/list][/list][/list]
        « Last Edit: December 27, 2018, 04:00:59 PM by JST200 » Logged
        JST200
        Jr. Member
        **
        Offline Offline

        Posts: 58


        « Reply #1 on: February 28, 2014, 11:13:48 AM »

        Hi Matteo,

        Thanks for testing for me - it's great to have someone else try it!

        It's not directly obvious from the crash codes (not to me at least!).
        It's possible that the when you transcoded from the code snippets something didn't copy properly - which was always my worry.

        So... could you send me a message and let me have your email address? I'll then send you a complete version of the eespvo files and perhaps you could send me your eespvo.ini and eespvo.log files.

        It would help if you could turn up the logging level too, to give me an idea of where it gets to before it crashes. To do so; create a file called eespvo.dbg and put the line "logLevel=5" in it.
        eespvo.dbg is read every 5secs, if it exists, to enable on the fly debug changes to made to the settings. It can contain all the same parameters as the .ini file. Once it is read it renames the file to eespvo.dbgR to stop it continually reading it. So if you want it to take affect again you have to rename it back to eespvo.dbg. You can disable the rename by using the parameter "dbgDsblDFRen=True" (hope that is all clear!)

        Rather than initially running eespvo.py at boot, it might be worth running it interactively - to iron out any problems. To do so;
        • edit eespvo.dbg and populate it with the following lines:
          • logLevel=5   #this increases the log level to debug mode
          • logConsole=True  #this outputs to the console as well as the default log file
          • flashLED=False  #this turns off LED flash mode. LED Flash mode requires root privileges
        • run Python3 and open eespvo.py
        • with eespvo.py open select Run > Run Module

        You could leave out the FlashLED setting if you run python3 from root (e.g. run LXterminal, then use "sudo python3 eespvo.py", but I wouldn't recommend it for now)

        Oh! If you are going to run it interactively, then it might be worth commenting out the line in /etc/rc.local so that if you reboot eespvo isn't auto run and confuses your testing.

        Postscript:
        This problem was caused by the eespvo.ini not being properly formed and eespvo.py not being resilient to some likely config errors in the file (e.g. the inclusion of spaces and " in defining the parameters). I have fixed the eespvo.py resilience issue in v3-03-07.
        « Last Edit: February 28, 2014, 04:04:19 PM by JST200 » Logged
        JST200
        Jr. Member
        **
        Offline Offline

        Posts: 58


        « Reply #2 on: February 28, 2014, 11:15:17 AM »

        After working with a couple of intrepid testers (thanks Matteo and Steve) I have realised that the version of eespvo.py that I posted above (v3-03-05) is quite sensitive to the format of the .ini file. So I have produced a version (v3-03-07) that is (hopefully) less so.

        Because it exceeds the characters allowed for a single post, I'm not sure that posting the code here is the most appropriate way to deal with updates. Therefore, if anyone wants a copy of the latest version then it's probably best if you send me a personal message and I'll be happy to oblige.

        Another thing to note with ref to smtp mail. If you are inside your smtp providers network (e.g. Virgin provides your broadband and is your email provider) you probably don't need to authenticate your smtp request. This is what the logfile will contain if the smtp fails with authentication errors:

        Quote
        20140227 175121 SMT > SMTP error, email not sent:
        20140227 175121 SMT > SMTP error: <class 'smtplib.SMTPAuthenticationError'>
        20140227 175121 SMT > SMTP error: (535, b'5.7.8 Error: authentication failed: another step is needed in authentication')
        20140227 175121 SMT > SMTP error: <traceback object at 0x16f3b48>

        So, if you do require smtp authentication put the following two lines in your eespvo.ini file:

        smtpUsernm=your smtp username
        smtpUserpw=your smtp password

         

        For anyone who tries eespvo.py, all improvement feedback, and even request, welcome!

        My future plans include:

        Cheers, Jim

        Postscript.
        I have realised I can attached the file eespvr.py as an attachment, for you to download, rather than try and insert it as "code". So I have attached the latest version v3-03-07 to my post above. However, it won't allow the upload of .py files, so I have had to rename it to a .txt file. Therefore you should rename eespvo v3-03-07.txt to eespvo.py after you download it.
        Logged
        JST200
        Jr. Member
        **
        Offline Offline

        Posts: 58


        « Reply #3 on: February 28, 2014, 11:17:01 AM »

        If it's of interest, here is what a typical email from eespvo.py looks like after a 15min network outage. No upload data was lost.

        If you only want the warnings and don't want the logfile detail to be added to the email, you can turn it off by putting the following parameter in the eespvo.ini file:

        emailLevel=0


        Quote
        eespvo (v3-03-07)
        ________________________________________

        The following warnings have been reported by eespvo:

        20140228 084043 Warning > Weather Update error, weather info ignored
        20140228 084123 Warning > Batch upload connect unsuccessful (network error likely)
        20140228 084543 Warning > Weather Update error, weather info ignored
        20140228 084623 Warning > Batch upload connect unsuccessful (network error likely)
        20140228 085043 Warning > Weather Update error, weather info ignored
        20140228 085123 Warning > Batch upload connect unsuccessful (network error likely)
        20140228 085543 Warning > Weather Update error, weather info ignored
        20140228 085623 Warning > Batch upload connect unsuccessful (network error likely)

        Logfile since initial warning:

        20140228 084043 Wea > Weather update error, weather info ignored
        20140228 084043 Wea > Error: <class 'urllib.error.URLError'>
        20140228 084043 Wea > Error: <urlopen error [Errno -2] Name or service not known>
        20140228 084043 Wea > Error: <traceback object at 0x1422620>
        20140228 084123 Bat > Batch upload unsuccessful with:
        20140228 084123 Bat > Connect error: <class 'socket.gaierror'>
        20140228 084123 Bat > Connect error: [Errno -2] Name or service not known
        20140228 084123 Bat > Connect error: <traceback object at 0x1422878>
        20140228 084543 Wea > Weather update error, weather info ignored
        20140228 084543 Wea > Error: <class 'urllib.error.URLError'>
        20140228 084543 Wea > Error: <urlopen error [Errno -2] Name or service not known>
        20140228 084543 Wea > Error: <traceback object at 0x1422b70>
        20140228 084623 Bat > Batch upload unsuccessful with:
        20140228 084623 Bat > Connect error: <class 'socket.gaierror'>
        20140228 084623 Bat > Connect error: [Errno -2] Name or service not known
        20140228 084623 Bat > Connect error: <traceback object at 0x1422d78>
        20140228 085043 Wea > Weather update error, weather info ignored
        20140228 085043 Wea > Error: <class 'urllib.error.URLError'>
        20140228 085043 Wea > Error: <urlopen error [Errno -2] Name or service not known>
        20140228 085043 Wea > Error: <traceback object at 0x1426120>
        20140228 085123 Bat > Batch upload unsuccessful with:
        20140228 085123 Bat > Connect error: <class 'socket.gaierror'>
        20140228 085123 Bat > Connect error: [Errno -2] Name or service not known
        20140228 085123 Bat > Connect error: <traceback object at 0x1426300>
        20140228 085543 Wea > Weather update error, weather info ignored
        20140228 085543 Wea > Error: <class 'urllib.error.URLError'>
        20140228 085543 Wea > Error: <urlopen error [Errno -2] Name or service not known>
        20140228 085543 Wea > Error: <traceback object at 0x1422968>
        20140228 085623 Bat > Batch upload unsuccessful with:
        20140228 085623 Bat > Connect error: <class 'socket.gaierror'>
        20140228 085623 Bat > Connect error: [Errno -2] Name or service not known
        20140228 085623 Bat > Connect error: <traceback object at 0x1422288>

        Cheers, Jim
        « Last Edit: February 28, 2014, 11:21:04 AM by JST200 » Logged
        oldman
        Newbie
        *
        Offline Offline

        Posts: 12


        « Reply #4 on: March 01, 2014, 10:03:54 AM »

        Hi Jim, Peter Longley says his Pi offering is due to go to testing next week, so still some time off being released.

        I have a Pi here doing nothing, so in basic laymans terms I need Pi, serial adaptor, wifi or wired connect, memory card, keyboard mouse & screen.
        Copy your code to a card, boot up, add stuff like pvoutput links etc and once its connected remove monitor keyboard and mouse?

        If you have time perhaps you could do a quick and dirty "how to" for us that cant Wink

        Thanks



        Logged

        3KW instal http://pvoutput.org/intraday.jsp?id=29060&sid=26619 by the seaside in Kent
        JST200
        Jr. Member
        **
        Offline Offline

        Posts: 58


        « Reply #5 on: March 01, 2014, 04:32:27 PM »

        Hi oldman (it doesn't seem right saying that somehow!  Undecided)

        I'm afraid any set up advice will have to be quick and dirty (hopefully not too much more so than you hoped) as I only have a few mins. Worse I am then on limited connectivity for a couple of weeks - though I'll help as best I can.

        There is loads of documentation on the web on how to do things Pi; so what I will do will is refer you to what I used and did, with some hints and tips along the way - if that's OK? The other constraint is that all my notes are based on my systems being Windows based. So if you are MAC oriented, I'm not sure how much different some of the setup below will be. Also, because I have always used Windows and have never been a fan of the UNIX bias towards incomprehensible command line structures, I use the Desktop interface and only drop in to LXTerminal to action command lines when I have to. Therefore, my setup notes are Pi Desktop biased.

        So first you need to choose your OS (of which there a quite a few!). I would recommend Raspbian as this was specifically designed for the Pi and is what I based my code on. You can install Raspian one of two ways. Either, use what's called the NOOB option, or install the OS yourself.

        For a raw beginner I would recommend the NOOB option. All you need to do is download the files from here: http://www.raspberrypi.org/downloads. Download the "NOOBS (offline and network install)" zip file, unpack it and copy files onto an SD Card.
        The same location provides the raw Raspbian image, but this requires the use of Win32DiskManager. This isn't difficult to use either, really.
        Either way just follow the instructions on the website.

        My process was to:
        • copy the NOOB files onto an SD Card
        • plug the Pi into my TV (the only device I have with HDMI)
        • plug in a mouse, keyboard and Ethernet (UTP) cable
        • Install Raspian
        • configure it to support VNC (both on Pi and PC)
        • unplug the mouse, keyboard and TV
        • reboot and connect to, now, headless Pi using VNC from PC

        The easiest way to understand how to do the above is refer to this website: http://learn.adafruit.com/category/learn-raspberry-pi.
        Follow:
        • Lesson 1 - preparing an SD Card for your Raspberry PI
        • Lesson 2 - first time configuration
        • Lesson 7 - remote control with VNC

        You might need to dip into "Lesson 3 - network setup"  to discover how to find out the Pi's IP address for the VNC step.

        These three (and a bit), lessons will give you step by step instructions on how to complete the above list.

        Of course, you could not bother with the VNC part, but I find it so much easier to just have my Pi next to my PC, with just power and Ethernet connected!

        One extra note... if you are using a VideoS video output (the yellow, round pin connector) then to get NOOB to display on your screen I think you have to press 1,2,3 or 4 on your keyboard (sorry I can't remember exactly which). By default it uses HDMI, but VideoS is supported.

        Assuming you, you now have your Pi up and running then I recommend adding Samba.
        Samba provides read/write access to the Pi file system from a Windows PC. I found it a real pain getting files onto the Pi until I discovered this facility. And it is pretty easy to configure. Here's a good website that gives you all you need to know; http://raspberrywebserver.com/serveradmin/share-your-raspberry-pis-files-and-folders-across-a-network.html

        You should now be in a position to follow my setup notes for my app in my first post above. You also should be in a good place to setup IanH64's code as an alternative (from here), should you wish.

        The only extra setup thing to add is; if you are using eespvo.ini to control the parameters for eespvo.py, be careful how you write them. If eespvo.py crashes and puts something like:

        Quote

        20140227 145921 Fatal: eespvo (ProcId: 2413 module: GeU) terminated with:
        20140227 145921 Fatal: <class 'UnicodeDecodeError'>
        20140227 145921 Fatal: 'ascii' codec can't decode byte 0xc2 in position 440: ordinal not in range(128)
        20140227 145921 Fatal: <traceback object at 0x9590f8>
        20140227 145921 Fatal: eespvo (ProcId: 2413) killed

        as the first entry in the logfile, then it is almost certain that it can't process the .ini file. I have tried to make it resilient to .ini errors, but it may not always be able to cope.
        You know if it has got past the .ini file if the 1st entry in the log file is:

        Quote
        20140225 120129 Sta > eespvo.py V3-03-07 [ProcID:nnnn] [PVSysId:nnnn]


        Finally... for what it's worth, here are all the "notes" I keep that remind me how to do stuff:

        Quote
        http://learn.adafruit.com/category/learn-raspberry-pi

        sudo raspi-config
             increase file space allocation
             change password for "pi" to "raspberry"

        reboot

        Install VNC Server
             sudo apt-get update
             sudo apt-get upgrade
             sudo apt-get install tightvncserver

        run vncserver to set password
             vncserver :1  ( Note 1 means it is using port 5001, note it is "<space> :1")

        reboot
        manually run vncserver and remote vnc in to check it works ok
             vncserver :1

        sudo raspi-config
             set to run to desktop

        reboot

        Set up vncserver for auto startup
             using LXTerminal or using startx >  File Manager
                  cd /home/pi
                  cd .config
                  mkdir autostart
                  cd autostart
                  nano tightvnc.desktop
                         [Desktop Entry]
                         Type=Application
                         Name=TightVNC
                         Exec=vncserver :1
                         StartupNotify=false

        using LXTerminal
             sudo raspi-config
                  set to run to desktop

        Autostart eespvo
             using LXTerminal
                  sudo nano /etc/rc.local
                       before exit() put:
                            (sleep 30;sudo /home/pi/eespvo/eespvo.scr)

        Copy eespvo folder to /home/pi/eespvo
        Make eespvo.scr executable using file manager > properties > permissions > make the file executable

        sudo kill <procid>


        To restore right click defaults to desktop
          pcmanfm --desktop-pref

        http://raspberrywebserver.com/serveradmin/share-your-raspberry-pis-files-and-folders-across-a-network.html

        sudo apt-get install samba samba-common-bin




        Once you have installed eespvo.py, and its associated files, then all you need to do is plus in the serial to usb cable, supplied with your Smart PV, and reboot you Pi. I plug the usb cable into the bottom usb port (the one nearest the pcb). There is no extra confi to do on the Smart PV. Hopefull, upon reboot, you should start to see the Pi's green ACT LED start to flash every 5s. Initially the flash may only be short, but after a while (once the Pi has settled down) the flash should last about 1s. When an upload is successful it will flash twice, every 5s, for about 30s. If an upload failure occurs it will flash 5 times, every 5s, for about 4m 30s

        Hope that all helps!

        Cheers, Jim
        « Last Edit: March 01, 2014, 05:02:57 PM by JST200 » Logged
        Pugwash
        Jr. Member
        **
        Offline Offline

        Posts: 62


        « Reply #6 on: March 01, 2014, 09:02:42 PM »

        Firstly great work Jim
        Having lots of bother trying to get samba up and running
        Is it possible to attach files eespvo.ini & eespvo.scr
        I have got ftp working and find this an easy method of transferring files
        once files in place I can edit them as needed

        Regards
        steve
        Logged

        Solar PV 4KW  16 x 250W Kinve panels Goodwe 3600 inverter Eco Eye Smart PV Monitor Solarimmersion mk3 http://pvoutput.org/intraday.jsp?id=20905&sid=18781
        JST200
        Jr. Member
        **
        Offline Offline

        Posts: 58


        « Reply #7 on: March 01, 2014, 09:42:08 PM »

        Hi Steve,

        Ok, so what I have done is completely restructured my first post above to attach three files to it:
        • eespvo v3-03-07_py.txt
        • eespvo_ini.txt
        • eespvo_scr.txt

        and then changed the instructions to reflect the new arrangement.
        These files need to be copied into "/home/pi/eespvo", renamed and edited in the case of eespvo.ini (as described in my first post)

        Since the second post, with the code snippets in is now redundant, I have deleted it.

        What problem are you having with samba? The trickiest bit is getting the samba.conf file modified correctly - if I remember correctly, I had a bit of a problem getting this right. Now I come to think of it, the order of the entries to samba.conf weren't quite right on the website. Anyway, this is what I added right at the end of samba.conf (I have included a few lines from above the addition to show where it goes in the file):

        Code:
        # The CD-ROM gets unmounted automatically after the connection to the
        #
        # If you don't want to use auto-mounting/unmounting make sure the CD
        # is mounted on /cdrom
        #
        ;   preexec = /bin/mount /cdrom
        ;   postexec = /bin/umount /cdrom


        [pihome]
        comment= Pi Home
        path=/home/pi
        only guest=no
        create mask=0777
        directory mask=0777
        public=no
        browseable=yes
        writeable=yes

        I've attached the full file, exactly as I use it, below. So copy on to your pi, rename it from "smb_cpnf.txt"  to "smb.conf"   and put it into folder "/etc". Then reboot.

        Then don't forget to set the password using an LXterminal session:

        smbpasswd -a pi

        It might be worth rebooting again at this point.

        RASPBERRYPI should then appear on your PC's file explorer like another network share. You may have to authenticate with; Account: pi (or possibly "pihome" you might have to try both as I can't remember which it is) and Password: what you set above.

        Does that help?

        Cheers, Jim.

        * smb_conf.txt (12.02 KB - downloaded 262 times.)
        « Last Edit: March 01, 2014, 10:45:49 PM by JST200 » Logged
        Pugwash
        Jr. Member
        **
        Offline Offline

        Posts: 62


        « Reply #8 on: March 01, 2014, 10:04:58 PM »

        That's a great help
        With samba I can get into the config file after editing I cannot save as I don't have permissions
        I didn't seem to be able to get to it via lxt
        I got to it through file manager its all still new to me so probably going about it wrong
        ill have a go at getting this up and running tomorrow

        Thanks again
        Logged

        Solar PV 4KW  16 x 250W Kinve panels Goodwe 3600 inverter Eco Eye Smart PV Monitor Solarimmersion mk3 http://pvoutput.org/intraday.jsp?id=20905&sid=18781
        JST200
        Jr. Member
        **
        Offline Offline

        Posts: 58


        « Reply #9 on: March 01, 2014, 10:22:17 PM »

        I think I was still re-editing my response when your saw it. I'm not sure if you saw the bit about smb.conf?

        Trying to remember now... I think the way I got write access to smb.conf was to:
        • run LXTerminal.
        • type "cd /etc/samba" to get you into the samba folder
        • type "sudo nano smb.conf" - sudo gives you root priv, nano is an editor (though not a great one, so you could try "sudo leafpad smb.conf")

        Something else to be aware of; when you connect to RASPBERRYPI you may see two folders: Pi and PiHome. Pi is Read Only and PiHome is Read/Write.

        Cheers, Jim
        « Last Edit: March 02, 2014, 08:37:03 AM by JST200 » Logged
        oldman
        Newbie
        *
        Offline Offline

        Posts: 12


        « Reply #10 on: March 02, 2014, 05:13:25 PM »

        Thanks very much Jim for the not so quick and dirty howto, that must have taken a while!

        Be prepared for further questions please  Smiley
        Logged

        3KW instal http://pvoutput.org/intraday.jsp?id=29060&sid=26619 by the seaside in Kent
        JST200
        Jr. Member
        **
        Offline Offline

        Posts: 58


        « Reply #11 on: March 16, 2014, 10:13:15 PM »

        Either:

        • potential testers recognised I was away and so didn't want to disturb me until I returned (thanks if so!) - possible  Smiley
        • have tried my program and don't like it - maybe   Sad
        • have tried it and it just works! - unlikely!   Shocked

        Anyway, I'm back online now, if anyone has any questions.

        Cheers, Jim

        Logged
        JST200
        Jr. Member
        **
        Offline Offline

        Posts: 58


        « Reply #12 on: March 20, 2014, 06:47:55 PM »

        New version added (v3-03-08).

        See first post...
        Logged
        CLIFFMCL
        Newbie
        *
        Offline Offline

        Posts: 3


        « Reply #13 on: April 08, 2015, 08:34:43 AM »

        Hi Jim

        Thanks for the script and info. I've sort of got it working on my system but would appreciate a couple of pointers to having it run better.

        I'm using the NOOBS download of Raspbian but the eespvo.scr file won't run. I set it (and the eespvo directory) to Read/Write/Execute for everyone but if I try to run it with a path (/home/pi/eespvo/eespov.scr) I get an error that the file doesn't exist. I get the same message if I execute the file from FileManager - where I can see it (see attached screen grab).

        After much messing around I ended up making a 'start' file which contains:

        cd /home/pi/eespvo
        sudo python3 eespvo.py

        and running it with a cron at boot. This means I don't have the loop resilience you mentioned. Is there something obvious the I'm missing?

        Secondly, I found the easiest way to get the files over was:
        a) if i wanted to edit them on the PC, save them to OneDrive, Google Drive or Dropbox and then log from the RPi web browser and download them
        b) to edit them on the RPi, run the browser to this post and download them directly.

        This saved messing with Samba at all.

        Thanks again


        * RPi.PNG (106.61 KB, 1047x821 - viewed 1569 times.)
        Logged
        JST200
        Jr. Member
        **
        Offline Offline

        Posts: 58


        « Reply #14 on: April 08, 2015, 09:14:02 AM »

        Hi Cliff,

        For the moment I am not quite sure why you are having the authentication problem.

        However, I should have re-posted an update to say that I have made quite a lot of progress with the app and it's setup.
        I have added new functionality/switches and made it almost bomb proof; my latest version has currently been running for 85days without missing a single 5min upload - even when PVOutput was out of action for about 6hours recently!

        If you would like a copy of the latest version, then PM me and we will work something out.

        Cheers, Jim
        Logged
        Pages: [1] 2 3 4 5 6   Go Up
          Print  
         
        Jump to:  

        Powered by MySQL Powered by PHP Powered by SMF 1.1.20 | SMF © 2013, Simple Machines
        SMFAds for Free Forums
        Simple Audio Video Embedder
        Valid XHTML 1.0! Valid CSS!