Stream becomes inactive regularly
Forum rules
This forum is run by the BATC (British Amateur Television Club), it is service made freely available to all interested parties, please do not abuse this privilege.
Thank you
This forum is run by the BATC (British Amateur Television Club), it is service made freely available to all interested parties, please do not abuse this privilege.
Thank you
Re: Stream becomes inactive regularly
Hi All
Just to let you know that we are trying to address this issue from 2 angles:
We are looking at the "new" BATC Server and trying to work out why it behaves differently from the old one.
I will be implementing a software triggered nightly reboot in the Portsdown Streamer.
Dave, G8GKQ
Just to let you know that we are trying to address this issue from 2 angles:
We are looking at the "new" BATC Server and trying to work out why it behaves differently from the old one.
I will be implementing a software triggered nightly reboot in the Portsdown Streamer.
Dave, G8GKQ
Re: Stream becomes inactive regularly
To help in our investigations, can anyone give us an exact date and time when the stream became inactive?
This would help enormously as initial investigations of the server logs do not show anything untoward.
Thanks
Dave, G8GKQ
This would help enormously as initial investigations of the server logs do not show anything untoward.
Thanks
Dave, G8GKQ
Re: Stream becomes inactive regularly
There are two listed in my post from August - is that too old now? If it is, I'll get the time and date from Monday...
73 Kay
73 Kay
Re: Stream becomes inactive regularly
We are continuing to investigate this problem:
Dave, G8GKQ
- I am running a modified Portsdown-based streamer continuously with full logging waiting for it to fail, so that we can check the diagnostic messages.
- I have just released a new Portsdown build that reboots the streamer early every morning at 0300 UTC so that the maximum outage period is 24 hours. Details here: viewtopic.php?f=69&t=5703
Dave, G8GKQ
Re: Stream becomes inactive regularly
Progress Report:
Having run a Portsdown streamer for a few days, I have had one drop-out; the streamer would have required a site visit to restart it. I have now issued a modification that reboots the streamer should this problem re-occur.
The modification is in Portsdown Stretch software update 201811030 and Portsdown Jessie software update 201811031. Both were issued this evening.
The instruction on how to apply this update on a running streamer are the same as before: viewtopic.php?f=69&t=5703#p17129 for Stretch, or here for Jessie: viewtopic.php?f=69&t=5704.
Please continue to report when you get streamer failures - we haven't got to the bottom of why Streaming PCs keep dropping out - if you could tell us the exact date time of any dropouts that you get that would help us enormously. The server logs are detailed and we are looking for needles in haystacks!
Dave, G8GKQ
Having run a Portsdown streamer for a few days, I have had one drop-out; the streamer would have required a site visit to restart it. I have now issued a modification that reboots the streamer should this problem re-occur.
The modification is in Portsdown Stretch software update 201811030 and Portsdown Jessie software update 201811031. Both were issued this evening.
The instruction on how to apply this update on a running streamer are the same as before: viewtopic.php?f=69&t=5703#p17129 for Stretch, or here for Jessie: viewtopic.php?f=69&t=5704.
Please continue to report when you get streamer failures - we haven't got to the bottom of why Streaming PCs keep dropping out - if you could tell us the exact date time of any dropouts that you get that would help us enormously. The server logs are detailed and we are looking for needles in haystacks!
Dave, G8GKQ
Re: Stream becomes inactive regularly
If it is happening after a day or a few days and the PI is on DHCP perhaps the router renegotiation is to blame or something is running out of memory, perhaps due to network delays etc. I take it this is not via WiFi but is a wired connection, the default password has been changed so it can't be easily hacked and that the network route or server doesn't re-negotiate.
Mike
Mike
Re: Stream becomes inactive regularly
Dave,
Thank you for your efforts on this, which is much appreciated. The worst part of this problem for repeater operators who stream from site is the need to go to site and reset the streamer. Since it is connected to the internet, it must be possible to access it remotely? Is there someone who understands the internet and networking who could devise a method to access the Pi remotely? It would then be easy to reboot and update it from home.
73 Shaun.
Thank you for your efforts on this, which is much appreciated. The worst part of this problem for repeater operators who stream from site is the need to go to site and reset the streamer. Since it is connected to the internet, it must be possible to access it remotely? Is there someone who understands the internet and networking who could devise a method to access the Pi remotely? It would then be easy to reboot and update it from home.
73 Shaun.
Re: Stream becomes inactive regularly
Hi
In reply to Mike:
This latest update was designed to overcome the "stream does not restart after router reboot" scenario. During one of my test router reboots, not only did it crash ffmpeg (which generates the stream), it crashed the (c) process that restarts ffmpeg when it crashes. So there is now another (bash) process that acts as a watchdog on the c process and reboots the RPi if that crashes. Hopefully the scheduled nightly reboot will overcome any dhcp issues.
In reply to Shaun:
Yes, you can reconfigure the on-site router to allow remote ssh access to the RPi. I did this at home last week to allow Evariste to access my development Portsdown to troubleshoot a tricky issue that he could not reproduce. What is required is to reconfigure the firewall in the on-site router to forward tcp ssh traffic from an external WAN port (I suggest a randomly chosen port, not the default) to port 22 on the Portsdown.
I have been hesitant to recommend this for repeater streaming up till now, because it introduces a potential attack vector (albeit one that we currently consider to be impenetrable) for hackers. However, it is a balance between the slightly increased risk and the convenience. I would welcome advice from others on the risk level.
This solution does of course rely on the fault not affecting network connectivity - I think that this would be a safe assumption from my tests so far.
I hope that helps to explain the current situation and options.
Dave, G8GKQ
In reply to Mike:
This latest update was designed to overcome the "stream does not restart after router reboot" scenario. During one of my test router reboots, not only did it crash ffmpeg (which generates the stream), it crashed the (c) process that restarts ffmpeg when it crashes. So there is now another (bash) process that acts as a watchdog on the c process and reboots the RPi if that crashes. Hopefully the scheduled nightly reboot will overcome any dhcp issues.
In reply to Shaun:
Yes, you can reconfigure the on-site router to allow remote ssh access to the RPi. I did this at home last week to allow Evariste to access my development Portsdown to troubleshoot a tricky issue that he could not reproduce. What is required is to reconfigure the firewall in the on-site router to forward tcp ssh traffic from an external WAN port (I suggest a randomly chosen port, not the default) to port 22 on the Portsdown.
I have been hesitant to recommend this for repeater streaming up till now, because it introduces a potential attack vector (albeit one that we currently consider to be impenetrable) for hackers. However, it is a balance between the slightly increased risk and the convenience. I would welcome advice from others on the risk level.
This solution does of course rely on the fault not affecting network connectivity - I think that this would be a safe assumption from my tests so far.
I hope that helps to explain the current situation and options.
Dave, G8GKQ
Re: Stream becomes inactive regularly
Hi Dave and Shaun
For remote management of Pis (although not so far a Portsdown one) I use a free service called remot3.it. The advantage over direct SSH is that you don't need to open ports on the firewall or have a static IP address. When you want to SSH to the Pi, you log into the remot3.it website which generates a one-time proxy SSH address. You SSH to that and it gets routed to your Pi. You do have to install a piece of client software on the Pi, though, and I haven't tried this with the Portsdown software.
73
Ray
For remote management of Pis (although not so far a Portsdown one) I use a free service called remot3.it. The advantage over direct SSH is that you don't need to open ports on the firewall or have a static IP address. When you want to SSH to the Pi, you log into the remot3.it website which generates a one-time proxy SSH address. You SSH to that and it gets routed to your Pi. You do have to install a piece of client software on the Pi, though, and I haven't tried this with the Portsdown software.
73
Ray
Re: Stream becomes inactive regularly
Hi Dave.
Since you have changed things to perform a 3am re-boot, there have been no failures from my end. The stream seems to carry on after your re-boot so the problem seems to have 'gone away!'. Its worked perfectly since.
I know this is a work-around, but it does prove that at least (in our case) its not the GB3YT streamer PC that fails - which it never did on the old system. The 3am re-start should not cause anyone in the UK a problem, but I know you have users outside the UK which may fall foul of the re-start?
Thanks for your hard work so far.
73 Kay, G8NZR
Since you have changed things to perform a 3am re-boot, there have been no failures from my end. The stream seems to carry on after your re-boot so the problem seems to have 'gone away!'. Its worked perfectly since.
I know this is a work-around, but it does prove that at least (in our case) its not the GB3YT streamer PC that fails - which it never did on the old system. The 3am re-start should not cause anyone in the UK a problem, but I know you have users outside the UK which may fall foul of the re-start?
Thanks for your hard work so far.
73 Kay, G8NZR