PDA

View Full Version : Improved GHL Profilux 3.1 stats and logs


StarF
01/29/2016, 04:51 AM
I have ben working on a small projekt to improve the graphs and log capabilities of my GHL Profilux 3.1. The system is running on a small rasberry pi 2, with a 800x600 touch screen display. The data is provided on a webserver, so the graphs can be acces from everywhere, as long you have acces to your pi. The idea is the make it easier to read the graphs, and have something that looks better than what GHL offer.. the profilux touch is ugly...

The system is working as it is now, but i was thinking if there were interest for it, to make it more userfriendly to set up, and publish it to free use? Right now you would have to edit some scripts in order to get the right sensor data. As ghl have removed the rss feed from the computer, it has also made it a bit more complicated to parse the data. But if there is interest i will finish it up, and publish it as gpl, probaly on github or something like that.

The system can run on anything that supports pythong 3. The website is fully responsive, making it adjust to the size of the screen that acces the site.


The overview, and test on colors warning/alarm
http://i.imgur.com/wDe1bMt.png

Ph graphs on chromebrowser
http://i.imgur.com/WW3k7Jb.jpg

Same graphs in a bigger screen with higher resolution.. (shows the responsive of the graphs.. its the same data that is loaded
http://i.imgur.com/qBsSd7g.jpg

same graphs on the rasberry pi screen.. again same data set.
http://i.imgur.com/tuxJaOu.jpg

The idea is to load 3 weeks of data into the graph, and then allow the user to zoom on the graphs to see more details, here is a picture where its zoomed..
http://i.imgur.com/uefRmbM.jpg

Redox graph on the pi
http://i.imgur.com/JR5YKv0.jpg

redox in chrome desktop computer
http://i.imgur.com/ce2sXPa.jpg



The graphs can be exported as png images, if you want. There is no limit on how much data you want to save. One day data, with samples every 5 min takes about 20kb of size on the system. (that is for 4 sensors)

If there is enought interest, i will make everything english, make it easy to add your probes, and remove the GHL part in the image... The project will be delived as a package, that can be run on anything that supports python 3.

doggydoc
01/29/2016, 06:31 AM
Definitely interested.

Halfmoon61
01/29/2016, 06:19 PM
Interested

GazzMan
01/30/2016, 04:00 AM
Interested in da uk lol

Diesel3443
02/01/2016, 12:08 PM
interested

StarF
02/02/2016, 03:08 AM
I am still working on a easy way to add the probe data, atm they are hardcoded into the system, i am working on a way to detect the probes and let the user deside what to be logged and shown... that is probaly the biggest isue. I think i have an idea on how to get direct acces to the ghl data, to allow live view of the data, but i need to get some time to test it.

The new overview shows the low, median and high endpoints in the data, over a given period.. atm its set for 7 days...

http://i.imgur.com/0NggN5M.png

doggydoc
02/02/2016, 06:31 AM
Very nice. Can't wait to see this in more depth

Diesel3443
02/02/2016, 08:44 AM
Agreed. looks great.

tegretol
02/02/2016, 03:19 PM
Interested

StarF
02/11/2016, 05:03 AM
I will probaly release a test version soon. This version will be locked to 4 sensors only.
Ph, Salt, redox and temp. The goal is to make it dynamic scaleable, but it will take some work.


This shows the development of the median/low and high values over time. The idea is, that through this view, you would get a good overview, in what range your systems runs. The median is the value that the system sits at the most, so if something happens you will start to see these graphs go down. So you know if you median ph value starts to get lower, you know something is wrong. There will be the same for the other sensors. (i am still working out how many data points there should be for this)
http://i.imgur.com/IIOA3G7.png

i am always open to iseas?

Do people need to log more than 4 sensors? or have more then 4 sensors shown?

doggydoc
02/11/2016, 06:29 AM
I have 5 sensors only because I keep a ph probe in my Ca Reactor.
But I think 4 is a great start.

StarF
02/11/2016, 06:39 AM
I have the same, but i dont want to log the data from the ca reactor.. it would be pointless , or ?

Diesel3443
02/11/2016, 09:58 AM
Only 4 here..

StarF
02/11/2016, 10:11 AM
Only 4 here..

what kind of probes?
ph, temp, con and redox?

Diesel3443
02/11/2016, 11:14 AM
Exactly.

ph, temp, con and redox

ballhog
02/16/2016, 01:22 PM
Cant wait, really cool stuff. Thank You

Madehtsobi
02/16/2016, 02:40 PM
Looks cool

I have 5 sensors

2 ph
Redox
Cond
Temp

StarF
02/17/2016, 02:00 AM
I have 3 things i need to fix, then its ready for testing...

StarF
02/17/2016, 02:01 AM
Looks cool

I have 5 sensors

2 ph
Redox
Cond
Temp

Do you want stats for all of them? i have 5 aswell, same as you but the one ph probe is for my calcium reactor, and i dont really want to see or track stats from that.

Madehtsobi
02/17/2016, 06:52 AM
Do you want stats for all of them? i have 5 aswell, same as you but the one ph probe is for my calcium reactor, and i dont really want to see or track stats from that.


Would b nice if you could! [emoji106]

StarF
02/17/2016, 08:20 AM
Would b nice if you could! [emoji106]

i would be possible in the future versions, the test version is locked for the first 4 sensors... the plan is to make it dynamic, so the user desides what to log, and not to log.. but that will take some work...

Madehtsobi
02/17/2016, 08:33 AM
Ok no problem! Still awesome thanks

unze
02/17/2016, 10:35 AM
The graphs look fantastic ,

GazzMan
02/17/2016, 01:23 PM
4 probes for me lol

StarF
02/18/2016, 02:46 AM
Getting a bit closer, still need to fix a few isues, but its close.

The requerements for running this is:

Python 3.5
the following python modules: requests, beautifulsoup4, lxml
Django 1.8

it ca run on anything that has Python 3.5 support, windows, linux, unix, macos..

guide:
Download and install python 3.5 for your system:
https://www.python.org/downloads

remember to click the option to add to the system path (if running windows)

when done start up a console, and write:

pip install request
pip install beautifulsoup4
pip install lxml (if you are using windows, and dont have a compile enviroment set up, you need to download a compiled version of lxml, and install that.. go to: http://www.lfd.uci.edu/~gohlke/pythonlibs/#lxml download the version for python 3.5 either 32bit if you are using 32bit python or 64bit use this command to install it "pip install lxml-3.5.0-cp35-none-win_amd64.whl")
pip install django==1.8.7 (should work with 1.9 but i havent tested it yet)

that is all, now the computer can run the enviroment..

The package i will be uploading in a day or so, should be able to run out of the box, when the above is done. Later on i will probaly make a install script.

Do people still want to test it? or have the above scared people away? ;)

here is a small update aswell, showing the new features.. (still taking ideas for features though)... I am trying to figure out how much data to keep, and to show.. I am curently thinking between 7 and 10 days for the main graphs, and mabye 14 days for the low/mid/high graphs, along with some historical data, mabye avarage pr month over a year or so....

Here you see around 10 days of log data. (remember its zoomable.. so you can zoom into the graps to show a section more detailed)

http://i.imgur.com/oBYcP8p.jpg

and zoomed in on a day

http://i.imgur.com/0RLykmm.jpg

The system records the lowest, and highest reading through out the day, and calcurates the median (the value the system sits at the most) and log these. This is done curently one hour a day. It takes all the recordings from today and 7 days back, then calcurates these values, and log them, this is done each hour. That means i end up with 12 recordings a day. These values are then added togerther, and i then show the avarage for each on a graphs pr day.. (the 7 day thing will probaly be cut down to 3 days or so..) i hope it makes sense..

http://i.imgur.com/tTFGyAp.jpg

you can remove a graph to make it easier to read the other.. here i have unselected the high graphs.

http://i.imgur.com/1kfi2Gd.jpg

and you can export...

http://i.imgur.com/TMV5U4w.jpg

doggydoc
02/18/2016, 06:32 AM
Wow.
Simply amazing. Wish I was more computer Savvy to really understand those instructions you explained for downloads.

StarF
02/19/2016, 06:54 PM
Wow.
Simply amazing. Wish I was more computer Savvy to really understand those instructions you explained for downloads.

it was only to get the framework requered to get it to work installed.. It will always be a requerement for now..

any way.. last few bugs fixed... let me know if any want to test it...

doggydoc
02/19/2016, 07:28 PM
I def want to test it.
Let me know what is needed to be done. Pm me if you want.

Diesel3443
02/19/2016, 10:05 PM
I want to test it as well

StarF
02/20/2016, 04:24 AM
i am interesting in seing how the date time behaves on a system running another time zone.

Have you both seen the requerements for getting the framework up and running? Do i need to make a more detailed guide?

doggydoc
02/20/2016, 02:06 PM
I def think a more detailed guide would be very practical

StarF
02/20/2016, 02:49 PM
I def think a more detailed guide would be very practical

i will think about how to make it easier to understand. But people shouldent expect a one click setup for this. It will requere at least some computer skills, involving knowing the ip address of the profilux 3, how to edit a config file. I will probaly make a one click install for the dependencies, when i have a more finished system. For now i was hoping that someone were interested and could follow the description on how to set up the framework to get it running, so i could see how the date data behaves in another timezone.

Diesel3443
02/20/2016, 11:11 PM
I will try and see what I can do tomorrow

Diesel3443
02/21/2016, 05:38 PM
i think I am ready to try in, I installed python and downloaded all the packages.

StarF
02/22/2016, 01:34 AM
i think I am ready to try in, I installed python and downloaded all the packages.

nice..

pm me your mail, and i will send it to you. I also need the ip address of your profilux 3 unit, so i can add that into the system.

StarF
04/30/2016, 04:16 AM
A short video of the system runnin on a raspberry 3 with touchscreen.

https://www.youtube.com/watch?v=aObLCClFxPQ

doggydoc
04/30/2016, 07:57 AM
That is amazing. I fed want to try it...
Just wish I was not comp illiterate!
Lol


Always at sea;

Dr. Antonio Longo

StarF
04/30/2016, 08:33 AM
That is amazing. I fed want to try it...
Just wish I was not comp illiterate!
Lol


Always at sea;

Dr. Antonio Longo

I have almost writen the entire system to make it easy to chose what to show and what not to show, so soon it should be able to be setup with out knowing much about computers... (if i get it finished that is)

i have considered making a hosted version, meaning that if you want to use it, it can be setup remotley, and it just need acces to your ghl system, and i will provide these graphs for you. I just know if its viable, running a small server for it, would requere some form off payment, although the system is not made for me to become rich, but just becaus i like to make things.

StarF
04/30/2016, 08:37 AM
oh btw.. its also possible to add small notes, and see how they affect the values. like if you add ph buffer. The note will be shown as a red dot, at the time it was created, and be shown on all graphs, this way it will be easy to see if a change, changes something across all graphs.

http://i.imgur.com/ANKIPFt.jpg

doggydoc
04/30/2016, 06:53 PM
I have almost writen the entire system to make it easy to chose what to show and what not to show, so soon it should be able to be setup with out knowing much about computers... (if i get it finished that is)



i have considered making a hosted version, meaning that if you want to use it, it can be setup remotley, and it just need acces to your ghl system, and i will provide these graphs for you. I just know if its viable, running a small server for it, would requere some form off payment, although the system is not made for me to become rich, but just becaus i like to make things.



Pm me or email me if you want to remotely do something. I am ok with fees.

Diesel3443
05/01/2016, 08:52 AM
Nice to see it progressing. I tried to get it up and running, but no luck. That I blame on me because I am not 100% computer savvy myself.

StarF
05/01/2016, 09:03 AM
Better get that hosted version working then, so it will be easy to setup.

GazzMan
05/01/2016, 02:20 PM
I'll go for this bud.

StarF
05/01/2016, 03:58 PM
i need a few ghl sources to be able to build this, so if people have their ghl units reacable from the internet, and they feel like helping. Let me know the ip address, then i will add it to my hosted system, and it will start to pull stats from it. When i have built up a website to handle the data, you will get acces to graphs similar to the ones shown here...

i have already built the backend, so the system can log diffrent ghl units correctly. there is just a whole lot of user stuff to do now..

Diesel3443
05/01/2016, 09:28 PM
You got pm


Sent from my iPhone using Tapatalk

StarF
05/05/2016, 04:59 AM
ca 40%færdig med den hosted løsning.

About 40% done with the hosted solution. I just need to add a few more features, then its all about finishing touches, error handeling and so on. The system is already collection data just fine, and are generated the graphs show in the thread here.

Diesel3443
05/05/2016, 08:21 AM
Awesome!

doggydoc
05/05/2016, 05:56 PM
You got pm



Always at sea;

Dr. Antonio Longo

StarF
05/16/2016, 10:38 AM
It now supports password protected ghl units. I do need a few to test with, to see if the data collection through a password protected ghl unit is reliable. Guest acces is fine.. pm me if you want to help test it...

i am considering making support for getting the data from myghl but i think it would be a feature for later. Right now i plan to finish this, with a few aditional functions and see how much interest there is. I dont plan on earning any money from this, but i expect it to be able to run cost free at least. So there will be a small fee for this, probaly around 1-2 doller pr month, it just has to be enought to cover the hosting fees.

next features will be, week and monthly stats.

Diesel3443
05/17/2016, 10:05 PM
You know I'm always in. Just let me know what I need to do


Sent from my iPhone using Tapatalk

StarF
05/18/2016, 02:39 AM
You know I'm always in. Just let me know what I need to do


Sent from my iPhone using Tapatalk

you should have acces though the url i sent on the mail a week or so ago? :)

Diesel3443
05/18/2016, 09:14 AM
I dont think I recieved it. Can you send it again?

StarF
05/18/2016, 09:54 AM
Odd... i have sent it as a pm now

StarF
05/24/2016, 05:30 AM
Now there is support for adding your test kit result.. I have support for: NO3,PO4,CA,KH,MG,NO2,HN4,SI,SR,I2,B,CU. If i missed any let me know.

doggydoc
05/24/2016, 06:00 AM
Wow that's awesome.
Looking great


Always at sea;

Dr. Antonio Longo

Diesel3443
05/24/2016, 12:26 PM
Looks great!

StarF
05/27/2016, 04:47 AM
Første del af de sidste funktioner er færdige.

Lav, median og højeste værdie i gennemsnit over uge vist i graf. Teorien er man på den måde kan se hvor stabilt ens akvarie køre hen over de forskelige uger. Der er pt kun 2 ugers data i systemet.. Men den skulle viste det hen over 7 uger. Der kommer en ligende funktion pr månede, hen over et helt år, og så pr år, når der er nok data i systemet til at vise det :)

First part of the last features on my list, for release is done. The week median data view. This shows low, median and high data avarage per week. The idea behind this, is you will be able to tell how stable your system is from week to week. right now the system shows 7 weeks, i will tweak this later.

after this, there will come a month view, that shows 12 months, and then a year view.. But that will come when i have enought data to work with.

You can remove parts of the data, so you can view low and high only, or low values through each week..

tegretol
05/28/2016, 03:36 AM
Hello
It's wonderfull
I'm very interesting for testing

Envoyé de mon SM-G935F en utilisant Tapatalk

StarF
05/28/2016, 03:45 AM
Hello
It's wonderfull
I'm very interesting for testing

Envoyé de mon SM-G935F en utilisant Tapatalk

No problem, pm me your mail, and the external address for your ghl profilux 3 webserver, and if it's protected with login the user and password. Guests access is enough, and I will create an account for you.

StarF
06/01/2016, 04:51 AM
I have myghl support now...

i do need at least 1 or 2 test persons for this, to see how stable it is.. I have tried to contact GHL about how they view remote acces to their myghl site, with out getting an answer, so they may close it down.

It will only capture the data every 10 minutes atm, its a bit more heavy to get the date this way, ghl have done everything they could, to make remote acces to their ghl data as hard to do as possible...

any way pm me your username and password for myghl, along with your email, if you want to test it.

StarF
08/08/2016, 02:27 AM
Stil looking for a few more testers...

davyboy1970
08/10/2016, 02:31 AM
Hello are you still looking for testers? can you pm me details please :)

StarF
08/10/2016, 02:32 AM
yes i am :) i need:
Name
Email
url/ip for Profilux acces
username and password if its protected by that.

StarF
09/17/2016, 04:32 AM
A small friday project of mine. :)

I make a small script, that allows data from my ghl unit to be inserted into influxdb (its a small fast database made for measurement data), that allowed me to setup grafana (its a graphing tool usaly used to monitor servers, switchers in bigger computer enviorements)

Grafana is a pretty neat system, when it comes to graphs. Currently i am readying data from my unit every 30 sec, and my graphs are updating live. I can set tresholds, and alarms, and easy select the range the data should be shown.

The script i made, allows the unit to be named, so if you want, you can have several units input data into the same database, and you can select the unit you want the data from. If you got more than one aquarium.


The script is writen in python 3..

http://i.imgur.com/BdlES7F.png

http://i.imgur.com/BdlES7F.png

http://i.imgur.com/BdlES7F.png

http://i.imgur.com/BdlES7F.png

http://grafana.org/
https://www.influxdata.com/

StarF
09/17/2016, 06:04 AM
i messed up the pictures...

http://i.imgur.com/BdlES7F.png

http://i.imgur.com/n8cPC9d.png

http://i.imgur.com/qhFIuUA.png

http://i.imgur.com/5zlUwxh.png

doggydoc
09/17/2016, 06:05 AM
Very very nice.


Always at sea;

Dr. Antonio Longo

gatordoc
09/17/2016, 12:57 PM
looks great! are you still interested in testers? Do you want myghl access?

StarF
09/17/2016, 12:59 PM
I gave up. Ghl did everything they could to make it impossible to get the data form myghl, and they don't seem interested in me extracting the data. I still have the system running with support for the built in webserver, so I could give you access to that if you want

gatordoc
09/17/2016, 03:04 PM
that would be great! only care about local access anyways...

StarF
09/17/2016, 03:06 PM
Pm me the external ip and user and password if required and I will set up your account

uFnEC
10/03/2016, 09:43 PM
Nice work!! could you share the source so I could host my own server?
Thanks

Diesel3443
12/02/2016, 09:36 AM
Looking awesome. I have had my tank down for a while and I am glad to see that you are still making improvements. Ill have my setup hooked up again in a few weeks.

StarF
02/16/2017, 04:43 AM
small update.. being redoing the entire system the last few month, to move it over to React, to allow me to do live polling of data from the ghl system. The idea is to make the dashboard update each 2/3 sec live, with out page reload..

I have also being trying out some diffrent ways to make the dashboard seem more modern.. here is a quick test i did.. i dont know if like those more than the squares....

StarF
07/20/2017, 06:53 AM
looking for testers again.

StarF
07/25/2017, 05:55 AM
I dont know if this has any interested to people or not.. but a small update:

the system now only uses the database for statestics, and is pulling the data live from the ghl unit.

http://i.imgur.com/Qiv02lm.jpg

1: this data is pulled from your ghl unit, and is updated every 5 sec
2: this data is pulled from the db and is updated every 30 minute.

http://i.imgur.com/8WCmHI0.jpg

settings that allow the user to change how offen updates are made.

it also allow to toggle if it should show alarms or not.. alarms are not shown yet, as i need to deside how it should show these...

as always.. testers are always needed...

reefwiser
07/25/2017, 07:23 AM
Really need to have you work for GHL :)

StarF
07/25/2017, 07:27 AM
Really need to have you work for GHL :)

well if they want that, they know where to find me ;)

Diesel3443
07/27/2017, 11:38 AM
looking for testers again.

I still want to test, for some reason I cant get the GHL to connect to it.

StarF
07/27/2017, 11:39 AM
I still want to test, for some reason I cant get the GHL to connect to it.

PM me the external IP of the unit and I will look into it