Stream becomes inactive regularly

Area for discussing streaming over the Internet, especially as it relates to ATV
Forum rules
This forum is run by the BATC (British Amateur Television Club), it is service made freely available to all interested parties, please do not abuse this privilege.

Thank you
G8GKQ
Site Admin
Posts: 2798
Joined: Sun Mar 22, 2009 1:21 pm

Re: Stream becomes inactive regularly

Post by G8GKQ » Wed Oct 17, 2018 8:40 am

Hi All

Just to let you know that we are trying to address this issue from 2 angles:

We are looking at the "new" BATC Server and trying to work out why it behaves differently from the old one.

I will be implementing a software triggered nightly reboot in the Portsdown Streamer.

Dave, G8GKQ

G8GKQ
Site Admin
Posts: 2798
Joined: Sun Mar 22, 2009 1:21 pm

Re: Stream becomes inactive regularly

Post by G8GKQ » Wed Oct 17, 2018 10:04 am

To help in our investigations, can anyone give us an exact date and time when the stream became inactive?

This would help enormously as initial investigations of the server logs do not show anything untoward.

Thanks

Dave, G8GKQ

G8NZR
Posts: 24
Joined: Mon Dec 16, 2013 9:49 am

Re: Stream becomes inactive regularly

Post by G8NZR » Wed Oct 17, 2018 10:24 am

There are two listed in my post from August - is that too old now? If it is, I'll get the time and date from Monday...

73 Kay

G8GKQ
Site Admin
Posts: 2798
Joined: Sun Mar 22, 2009 1:21 pm

Re: Stream becomes inactive regularly

Post by G8GKQ » Wed Oct 31, 2018 1:47 pm

We are continuing to investigate this problem:
  • I am running a modified Portsdown-based streamer continuously with full logging waiting for it to fail, so that we can check the diagnostic messages.
  • I have just released a new Portsdown build that reboots the streamer early every morning at 0300 UTC so that the maximum outage period is 24 hours. Details here: viewtopic.php?f=69&t=5703
Investigations of the streaming server logs have so far been inconclusive.

Dave, G8GKQ

G8GKQ
Site Admin
Posts: 2798
Joined: Sun Mar 22, 2009 1:21 pm

Re: Stream becomes inactive regularly

Post by G8GKQ » Sat Nov 03, 2018 11:13 pm

Progress Report:

Having run a Portsdown streamer for a few days, I have had one drop-out; the streamer would have required a site visit to restart it. I have now issued a modification that reboots the streamer should this problem re-occur.

The modification is in Portsdown Stretch software update 201811030 and Portsdown Jessie software update 201811031. Both were issued this evening.

The instruction on how to apply this update on a running streamer are the same as before: viewtopic.php?f=69&t=5703#p17129 for Stretch, or here for Jessie: viewtopic.php?f=69&t=5704.

Please continue to report when you get streamer failures - we haven't got to the bottom of why Streaming PCs keep dropping out - if you could tell us the exact date time of any dropouts that you get that would help us enormously. The server logs are detailed and we are looking for needles in haystacks!

Dave, G8GKQ

g0mjw
Posts: 2330
Joined: Sat Sep 20, 2014 9:15 am

Re: Stream becomes inactive regularly

Post by g0mjw » Sun Nov 04, 2018 8:27 am

If it is happening after a day or a few days and the PI is on DHCP perhaps the router renegotiation is to blame or something is running out of memory, perhaps due to network delays etc. I take it this is not via WiFi but is a wired connection, the default password has been changed so it can't be easily hacked and that the network route or server doesn't re-negotiate.

Mike

g8vpg
Posts: 381
Joined: Fri Oct 11, 2013 9:35 am

Re: Stream becomes inactive regularly

Post by g8vpg » Sun Nov 04, 2018 10:50 am

Dave,
Thank you for your efforts on this, which is much appreciated. The worst part of this problem for repeater operators who stream from site is the need to go to site and reset the streamer. Since it is connected to the internet, it must be possible to access it remotely? Is there someone who understands the internet and networking who could devise a method to access the Pi remotely? It would then be easy to reboot and update it from home.
73 Shaun.

G8GKQ
Site Admin
Posts: 2798
Joined: Sun Mar 22, 2009 1:21 pm

Re: Stream becomes inactive regularly

Post by G8GKQ » Sun Nov 04, 2018 12:16 pm

Hi

In reply to Mike:

This latest update was designed to overcome the "stream does not restart after router reboot" scenario. During one of my test router reboots, not only did it crash ffmpeg (which generates the stream), it crashed the (c) process that restarts ffmpeg when it crashes. So there is now another (bash) process that acts as a watchdog on the c process and reboots the RPi if that crashes. Hopefully the scheduled nightly reboot will overcome any dhcp issues.

In reply to Shaun:

Yes, you can reconfigure the on-site router to allow remote ssh access to the RPi. I did this at home last week to allow Evariste to access my development Portsdown to troubleshoot a tricky issue that he could not reproduce. What is required is to reconfigure the firewall in the on-site router to forward tcp ssh traffic from an external WAN port (I suggest a randomly chosen port, not the default) to port 22 on the Portsdown.

I have been hesitant to recommend this for repeater streaming up till now, because it introduces a potential attack vector (albeit one that we currently consider to be impenetrable) for hackers. However, it is a balance between the slightly increased risk and the convenience. I would welcome advice from others on the risk level.

This solution does of course rely on the fault not affecting network connectivity - I think that this would be a safe assumption from my tests so far.

I hope that helps to explain the current situation and options.

Dave, G8GKQ

M0DHP
Posts: 92
Joined: Sun Sep 24, 2017 3:25 pm

Re: Stream becomes inactive regularly

Post by M0DHP » Sun Nov 04, 2018 12:23 pm

Hi Dave and Shaun

For remote management of Pis (although not so far a Portsdown one) I use a free service called remot3.it. The advantage over direct SSH is that you don't need to open ports on the firewall or have a static IP address. When you want to SSH to the Pi, you log into the remot3.it website which generates a one-time proxy SSH address. You SSH to that and it gets routed to your Pi. You do have to install a piece of client software on the Pi, though, and I haven't tried this with the Portsdown software.

73
Ray

G8NZR
Posts: 24
Joined: Mon Dec 16, 2013 9:49 am

Re: Stream becomes inactive regularly

Post by G8NZR » Thu Nov 08, 2018 9:56 am

Hi Dave.

Since you have changed things to perform a 3am re-boot, there have been no failures from my end. The stream seems to carry on after your re-boot so the problem seems to have 'gone away!'. Its worked perfectly since.

I know this is a work-around, but it does prove that at least (in our case) its not the GB3YT streamer PC that fails - which it never did on the old system. The 3am re-start should not cause anyone in the UK a problem, but I know you have users outside the UK which may fall foul of the re-start?

Thanks for your hard work so far.

73 Kay, G8NZR

Post Reply

Return to “Streaming”