Help!: Apex failure = crash in progress

entropygain · 11/11/2018, 03:11 PM

I'd greatly appreciate the help of the community on managing a crash in progress.

240g new reef tank with corals and a few small fish transferred from old 140g reef tank transferred 2-3 weeks ago. All stable, perfect parameters for 2 weeks.
New system was upgraded to Apex system controls, including control for two heaters. The tank is on a backup power system in an office setting.

04:00 yesterday
--the power went off and generators kicked in.
--According to APEx history tank temp started to climb shortly thereafter.
--No alarms received.
--Tank reached 95F by 16:00 and climbed slowly to 96F

11:45 this morning the problem was discovered by our facilities manager.
--Apex was off-line (though might have been our routers) but apparently powered on and locked the heaters on.

12:00 today rushed to office
-- unplugged heaters
--added ice to sump, put fan on surface
-- temp back to 90 and falling

Preparing for water change and running carbon
Haven't had a chance to test parameters, but who cares if the temp is 95

Any other suggestions? Maybe completely cooked but would like to salvage what I can.

thanks!

CAPT_Dave · 11/11/2018, 03:45 PM

Oxygen solubility is inversely proportional to temperature. That means the warmer the water the less oxygen. Make sure you're aerating. If you have a skimmer that should do but if not direct your powerheads at the surface to get more churn.

Sent from my iPad using Tapatalk

entropygain · 11/11/2018, 04:34 PM

Dave - thx
Yes Skimmer + 2 power heads + return gives good aeration

I added zeolites in media bags to help blunt any ammonia spike, but don't know how well they might work. The water is somewhat cloudy so I presume bacterial bloom and skimmer is going nuts. Hopefully, this will limit the spike.

Temp down to 85 and falling.

scuzy · 11/11/2018, 04:45 PM

On your heaters do you have a secondary controller like a ranco or built in thermistor? This will prevent issues like this from happening. Also what's you apex statement for the heaters?

I run a ranco with apex as backup. The ranco controls the heaters and the Apex will shut the plug if tempgoes whacked.

Sent from my iPhone using Tapatalk Pro

entropygain · 11/11/2018, 05:02 PM

Thx Scuzy

I screwed up and didn't have a backup controller on the heaters. My bad! Let Murphy have an angle and he bites you in the butt.

but I still don't understand why the Apex didn't shut them down.

I don't understand what you mean by apex statement. Unfortunately, I didn't have the alarm configured properly, but it still should have shut off the outlet which was set for 78-79.5 with a fallback of OFF

scuzy · 11/11/2018, 05:20 PM

Ok wanted to see your fallback statement. That looks correct unless your probe want sending the right temperature reading.

Sent from my iPhone using Tapatalk Pro

entropygain · 11/11/2018, 05:29 PM

When I rebooted the Apex the history was horrible:

The tank looks terrible with huge sinularia shrunk and shedding slime, all my hammers completely retracted.

ca1ore · 11/11/2018, 05:46 PM

One thing to check is whether apex created a new address for the power bar. I had this happen with the new apex when the brain lost power. Upon restoration it had re-addressed the power bar and the heater outlets were set to ON rather than AUTO. I use EJ with built in thermostats so cannot cook the tank. I now run the apex brain off a UPS so it never loses power.

entropygain · 11/11/2018, 06:03 PM

Partial apologies to Apex.

Everything is on backup power which kicks in a few seconds after a power loss (but not un-interruptable). When the backup power kicked in, the power bar didn't restore. It should have.

However, the overheat isn't Apex fault. Apparently, my facilities crew responding to the power outage found the power bar unlit and the tank inactive. To fix it they unplugged everything from the Apex and plugged it into the wall - including heaters! Obviously my crew training was inadequate. They were afraid to call me in the middle of the night and should have just left it alone...

Still a sad outcome

IUfan · 11/11/2018, 08:44 PM

Sorry to hear this happened. You should connect to apex fusion and turn Heartbeat ON. Then as soon as the apex loses connectivity to fusion (power failure, router failure etc.) it triggers an alarm and an email/text alert.

Sometimes it’s just the router resetting, sometimes it’s a power failure. Either way, good to know it via an email alert.

Sent from my iPhone using Tapatalk

scuzy · 11/11/2018, 08:58 PM

Quote:

Originally Posted by ca1ore

One thing to check is whether apex created a new address for the power bar. I had this happen with the new apex when the brain lost power. Upon restoration it had re-addressed the power bar and the heater outlets were set to ON rather than AUTO. I use EJ with built in thermostats so cannot cook the tank. I now run the apex brain off a UPS so it never loses power.

I had an issue with my fmk it somehow added a new one and it added 1 gallon of kalk into my tank.

Sent from my iPhone using Tapatalk Pro

Im14abeer · 11/12/2018, 07:59 AM

For future reference, the thermostat on your heater should be set one degree higher than your controller parameter. This way if you lose control the heater will still shut itself off before things get out of hand. It's nice that heaters have this built in redundancy. It won't help if the contact welds itself closed, but it would have prevented this particular situation. Hope everything recovers for you, this hobby is a constant battle against Murphy.

11/11/2018, 03:11 PM	#1
entropygain Registered Member Join Date: Mar 2002 Location: San Diego Posts: 283	Help!: Apex failure = crash in progress I'd greatly appreciate the help of the community on managing a crash in progress. 240g new reef tank with corals and a few small fish transferred from old 140g reef tank transferred 2-3 weeks ago. All stable, perfect parameters for 2 weeks. New system was upgraded to Apex system controls, including control for two heaters. The tank is on a backup power system in an office setting. 04:00 yesterday --the power went off and generators kicked in. --According to APEx history tank temp started to climb shortly thereafter. --No alarms received. --Tank reached 95F by 16:00 and climbed slowly to 96F 11:45 this morning the problem was discovered by our facilities manager. --Apex was off-line (though might have been our routers) but apparently powered on and locked the heaters on. 12:00 today rushed to office -- unplugged heaters --added ice to sump, put fan on surface -- temp back to 90 and falling Preparing for water change and running carbon Haven't had a chance to test parameters, but who cares if the temp is 95 Any other suggestions? Maybe completely cooked but would like to salvage what I can. thanks! __________________ ...remind me to go diving more often Current Tank Info: 140 gal LPS/Softie reef

11/11/2018, 03:45 PM	#2
CAPT_Dave Registered Member Join Date: Feb 2015 Location: San Diego Posts: 157	Oxygen solubility is inversely proportional to temperature. That means the warmer the water the less oxygen. Make sure you're aerating. If you have a skimmer that should do but if not direct your powerheads at the surface to get more churn. Sent from my iPad using Tapatalk __________________ Dave Current Tank Info: 120 g mixed reef

11/11/2018, 04:34 PM	#3
entropygain Registered Member Join Date: Mar 2002 Location: San Diego Posts: 283	Dave - thx Yes Skimmer + 2 power heads + return gives good aeration I added zeolites in media bags to help blunt any ammonia spike, but don't know how well they might work. The water is somewhat cloudy so I presume bacterial bloom and skimmer is going nuts. Hopefully, this will limit the spike. Temp down to 85 and falling. __________________ ...remind me to go diving more often Current Tank Info: 140 gal LPS/Softie reef

11/11/2018, 05:02 PM	#5
entropygain Registered Member Join Date: Mar 2002 Location: San Diego Posts: 283	Thx Scuzy I screwed up and didn't have a backup controller on the heaters. My bad! Let Murphy have an angle and he bites you in the butt. but I still don't understand why the Apex didn't shut them down. I don't understand what you mean by apex statement. Unfortunately, I didn't have the alarm configured properly, but it still should have shut off the outlet which was set for 78-79.5 with a fallback of OFF __________________ ...remind me to go diving more often Current Tank Info: 140 gal LPS/Softie reef

11/11/2018, 05:29 PM	#7
entropygain Registered Member Join Date: Mar 2002 Location: San Diego Posts: 283	When I rebooted the Apex the history was horrible: The tank looks terrible with huge sinularia shrunk and shedding slime, all my hammers completely retracted. __________________ ...remind me to go diving more often Current Tank Info: 140 gal LPS/Softie reef

11/11/2018, 04:45 PM	#4
scuzy Registered Member Join Date: Sep 2003 Location: San Jose, CA Posts: 6,499	On your heaters do you have a secondary controller like a ranco or built in thermistor? This will prevent issues like this from happening. Also what's you apex statement for the heaters? I run a ranco with apex as backup. The ranco controls the heaters and the Apex will shut the plug if tempgoes whacked. Sent from my iPhone using Tapatalk Pro

11/11/2018, 05:20 PM	#6
scuzy Registered Member Join Date: Sep 2003 Location: San Jose, CA Posts: 6,499	Ok wanted to see your fallback statement. That looks correct unless your probe want sending the right temperature reading. Sent from my iPhone using Tapatalk Pro

11/11/2018, 05:46 PM	#8
ca1ore Grizzled & Cynical Join Date: Apr 2006 Location: Stamford, CT Posts: 17,319	One thing to check is whether apex created a new address for the power bar. I had this happen with the new apex when the brain lost power. Upon restoration it had re-addressed the power bar and the heater outlets were set to ON rather than AUTO. I use EJ with built in thermostats so cannot cook the tank. I now run the apex brain off a UPS so it never loses power. __________________ Simon Got back into the hobby ..... planned to keep it simple ..... yeah, right ..... clearly I need a new plan! Pet peeve: anemones host clowns; clowns do not host anemones! Current Tank Info: 450 Reef; 120 refugium; 60 Frag Tank, 30 Introduction tank; multiple QTs

11/11/2018, 06:03 PM	#9
entropygain Registered Member Join Date: Mar 2002 Location: San Diego Posts: 283	Partial apologies to Apex. Everything is on backup power which kicks in a few seconds after a power loss (but not un-interruptable). When the backup power kicked in, the power bar didn't restore. It should have. However, the overheat isn't Apex fault. Apparently, my facilities crew responding to the power outage found the power bar unlit and the tank inactive. To fix it they unplugged everything from the Apex and plugged it into the wall - including heaters! Obviously my crew training was inadequate. They were afraid to call me in the middle of the night and should have just left it alone... Still a sad outcome __________________ ...remind me to go diving more often Current Tank Info: 140 gal LPS/Softie reef

11/11/2018, 08:44 PM	#10
IUfan Registered Member Join Date: Nov 2012 Posts: 841	Sorry to hear this happened. You should connect to apex fusion and turn Heartbeat ON. Then as soon as the apex loses connectivity to fusion (power failure, router failure etc.) it triggers an alarm and an email/text alert. Sometimes it’s just the router resetting, sometimes it’s a power failure. Either way, good to know it via an email alert. Sent from my iPhone using Tapatalk

11/12/2018, 07:59 AM	#12
Im14abeer BRILLIANT! Join Date: Jan 2005 Location: Grosse Pointe Farms, MI Posts: 406	For future reference, the thermostat on your heater should be set one degree higher than your controller parameter. This way if you lose control the heater will still shut itself off before things get out of hand. It's nice that heaters have this built in redundancy. It won't help if the contact welds itself closed, but it would have prevented this particular situation. Hope everything recovers for you, this hobby is a constant battle against Murphy. __________________ Veni, vidi, fishi Current Tank Info: Mixed 235 "reefin' in the corner." Reefbreeders 48 & 32. 2x Tunze 6255 on a 7096, Basement sump, Avast cs1, mag 18 return. Chaeto, kalk.