Critical issue 5.1.2 details

[Thierry Quemerais]

Dear Multitech Team,

We deployed a lot of v5.1.2 multitech gateways and according to your last release note we understand there is a critical issue with this version which make the LoRa Packet Forwarder unresponsive in somes conditions.

We would have more details regarding this issue in order to prepare a risk analysis for our customers. So we would like to know :

– What is the reproduce steps for this issue ?
– What is the reproduce rate ?
– Is there any workaround like restarting the gateway once every day ?
– Does this issue happen when the pkt_fwd process is restarted every day ?
– Can we disable pkt fwd log to avoid this issue (Remove “-l $pkt_fwd_log” from lora-network-server)

All information will be usefull to properly understand the problem and provide an accurate risk analysis to ours customers.

Thank you,
Best regards,
Thierry

[Jason Reiss]

– What is the reproduce steps for this issue ?
Configure the Conduit in lora packet forwarder mode.
After /var/log/lora-pkt-fwd-1.log file grows to 512K logrotate will try to copy the file, create a compressed version and HUP the processes.
The logrotate script fails as the lora-network-server is not running. This leaves a lora-pkt-fwd-1.log.1 file left uncompressed. The next time logrotate runs it will quit because this file exists. The original lora-pkt-fwd-1.log file will continue to grow taking up RAM as the log directory is in a RAM disk.

– What is the reproduce rate ?
100% of MTCDT/MTCAP configured in packet forwarder mode

– Is there any workaround like restarting the gateway once every day ?
Yes restarting the gateway will clear all log files

– Does this issue happen when the pkt_fwd process is restarted every day ?
Restarting the pkt_fwd will not clear the log files.

– Can we disable pkt fwd log to avoid this issue (Remove “-l $pkt_fwd_log” from lora-network-server)
Yes this would avoid the issue. Otherwise an IPK can be installed to correct the logrotate scripts a link is provided on the changelog downloads page.

https://www.multitech.net/developer/software/aep/aep-firmware-changelog/

[Thierry Quemerais]

Thank you for this clear and detailled reply, It’s very helpfull.

Best regards,
Thierry

[Michelle Fitzgerald]

Hi,
Came across this issue on a gateway, where the packet forwarder logs filled up disk space – implemented the fix above, and although it does rotate the logs, when the packet forwarder is restarted it doesn’t generate any further logs. Omitting the packet forwarder restart from /etc/logrotate.d/lora-logging.conf file seems to fix this – is a restart of the packet forwarder a requirement ?

Thanks,
Michelle

[Jason Reiss]

Michelle,

Could you please provide more details?
What fix did you implement?
What version are you running? $ cat /etc/issue
When is the packet forwarder restarted?

[Michelle Fitzgerald]

Hi Jason,

I added the following to /etc/logrotate.d/lora-logging.conf (taken from

/var/log/lora-*.log {
size 512k
rotate 4
compress
copytruncate
missingok
postrotate
set -e;
/usr/bin/killall -HUP lora_pkt_fwd || true;
/usr/bin/killall -HUP lora-network-server || true;
endscript
}

While it does rotate the log files, once the packet forwarder restarts it does not log to /var/log/lora-pkt-fwd-1.log. The gw is running 1.7.4.

regards,
Michelle

[Jason Reiss]

The fix applies to mPower 5.1.2 only.
On mPower 1.7.4 the HUP will restart the packet forwarder.

By default there is no PKF logs in mPower 1.7.4
Have you installed packages or modified the init scripts to create PKF logs?

[Michelle Fitzgerald]

Hi Jason,

Yes I have modified the init script to create PKF logs. If I just leave the lora-logging.conf as follows all works :

/var/log/lora-*.log {
size 512k
rotate 4
compress
copytruncate
missingok
postrotate

}

Is a restart of the packet forwarder required – normally we would not restart unless there was a configuration change, or loss of power.

Thanks,
Michelle

[Jason Reiss]

Yes I have modified the init script to create PKF logs.

How did you implement this?

Redirecting the stdout/stderr to a file with the > or >> does not work with logrotate. The original log file continues to grow as the file cursor does not return to 0 after the truncation. The start of the file is filled with nulls.

In 5.1.2 we added logging capability to the PKF to handle the rotation correctly.

[Michelle Fitzgerald]

Hi Jason,

Yes, its through redirect – I guess I have a workaround on this in the current setup, so will monitor and see how performs.

Thanks for your help,
Michelle

[Author]

Posts