Critical issue 5.1.2 details
Home › Forums › Conduit: AEP Model › Critical issue 5.1.2 details
Tagged: log rotate 5.1.2 critical issue
- This topic has 9 replies, 3 voices, and was last updated 5 years ago by
Michelle Fitzgerald.
-
AuthorPosts
-
March 19, 2020 at 3:34 am #30454
Thierry Quemerais
ParticipantDear Multitech Team,
We deployed a lot of v5.1.2 multitech gateways and according to your last release note we understand there is a critical issue with this version which make the LoRa Packet Forwarder unresponsive in somes conditions.
We would have more details regarding this issue in order to prepare a risk analysis for our customers. So we would like to know :
– What is the reproduce steps for this issue ?
– What is the reproduce rate ?
– Is there any workaround like restarting the gateway once every day ?
– Does this issue happen when the pkt_fwd process is restarted every day ?
– Can we disable pkt fwd log to avoid this issue (Remove “-l $pkt_fwd_log” from lora-network-server)All information will be usefull to properly understand the problem and provide an accurate risk analysis to ours customers.
Thank you,
Best regards,
ThierryMarch 19, 2020 at 9:15 am #30458Jason Reiss
Keymaster– What is the reproduce steps for this issue ?
Configure the Conduit in lora packet forwarder mode.
After /var/log/lora-pkt-fwd-1.log file grows to 512K logrotate will try to copy the file, create a compressed version and HUP the processes.
The logrotate script fails as the lora-network-server is not running. This leaves a lora-pkt-fwd-1.log.1 file left uncompressed. The next time logrotate runs it will quit because this file exists. The original lora-pkt-fwd-1.log file will continue to grow taking up RAM as the log directory is in a RAM disk.– What is the reproduce rate ?
100% of MTCDT/MTCAP configured in packet forwarder mode– Is there any workaround like restarting the gateway once every day ?
Yes restarting the gateway will clear all log files– Does this issue happen when the pkt_fwd process is restarted every day ?
Restarting the pkt_fwd will not clear the log files.– Can we disable pkt fwd log to avoid this issue (Remove “-l $pkt_fwd_log” from lora-network-server)
Yes this would avoid the issue. Otherwise an IPK can be installed to correct the logrotate scripts a link is provided on the changelog downloads page.March 19, 2020 at 9:17 am #30459Thierry Quemerais
ParticipantThank you for this clear and detailled reply, It’s very helpfull.
Best regards,
ThierryApril 6, 2020 at 5:20 am #30510Michelle Fitzgerald
ParticipantHi,
Came across this issue on a gateway, where the packet forwarder logs filled up disk space – implemented the fix above, and although it does rotate the logs, when the packet forwarder is restarted it doesn’t generate any further logs. Omitting the packet forwarder restart from /etc/logrotate.d/lora-logging.conf file seems to fix this – is a restart of the packet forwarder a requirement ?Thanks,
MichelleApril 6, 2020 at 8:26 am #30511Jason Reiss
KeymasterMichelle,
Could you please provide more details?
What fix did you implement?
What version are you running? $ cat /etc/issue
When is the packet forwarder restarted?April 6, 2020 at 9:26 am #30512Michelle Fitzgerald
ParticipantHi Jason,
I added the following to /etc/logrotate.d/lora-logging.conf (taken from
/var/log/lora-*.log {
size 512k
rotate 4
compress
copytruncate
missingok
postrotate
set -e;
/usr/bin/killall -HUP lora_pkt_fwd || true;
/usr/bin/killall -HUP lora-network-server || true;
endscript
}While it does rotate the log files, once the packet forwarder restarts it does not log to /var/log/lora-pkt-fwd-1.log. The gw is running 1.7.4.
regards,
MichelleApril 6, 2020 at 9:39 am #30514Jason Reiss
KeymasterThe fix applies to mPower 5.1.2 only.
On mPower 1.7.4 the HUP will restart the packet forwarder.By default there is no PKF logs in mPower 1.7.4
Have you installed packages or modified the init scripts to create PKF logs?April 6, 2020 at 10:35 am #30515Michelle Fitzgerald
ParticipantHi Jason,
Yes I have modified the init script to create PKF logs. If I just leave the lora-logging.conf as follows all works :
/var/log/lora-*.log {
size 512k
rotate 4
compress
copytruncate
missingok
postrotate}
Is a restart of the packet forwarder required – normally we would not restart unless there was a configuration change, or loss of power.
Thanks,
MichelleApril 6, 2020 at 11:06 am #30516Jason Reiss
KeymasterYes I have modified the init script to create PKF logs.
How did you implement this?
Redirecting the stdout/stderr to a file with the > or >> does not work with logrotate. The original log file continues to grow as the file cursor does not return to 0 after the truncation. The start of the file is filled with nulls.
In 5.1.2 we added logging capability to the PKF to handle the rotation correctly.
April 6, 2020 at 11:41 am #30517Michelle Fitzgerald
ParticipantHi Jason,
Yes, its through redirect – I guess I have a workaround on this in the current setup, so will monitor and see how performs.
Thanks for your help,
Michelle -
AuthorPosts
- You must be logged in to reply to this topic.