mDot running out of memory

[Mark Makarychev]

I’m using the ARM mbed IDE to code my mDot, and in my project I am using libmDot-mbed5 v2.0.15 with mbed-os v5.1.5. I was content with how my code worked and decided to run some endurance tests. The tests revealed the following bug: after sending around 800 frames (FCnt varies from 2af to 588 in various tests) with 13 bytes of payload and ACKs set to 1, I always get the following error just before sending:
Operator new[] out of memory
after which the mDot stops.

I started checking whether my payload vector of uint8_t might be the cause, but it was not. Why I’m sure: I clear it before each send, even tried using std::vector<uint8_t>(payload).swap(payload); (error still persists), and a final test of looping my send method 1000+ times without actually sending anything gave no errors.

So my question is as follows:
Does libmDot store data that is sent or received (frames) in some vector? If not, what else could be the cause of the mDot running out of memory? The only way out right now seems to do a soft reset every 600 or so frames, but this isn’t really a good solution =/.

mDot output:
Just before the error:

[INFO] Configure radio for TX
[INFO] Time-on-air for 13 byte: 164
[INFO] Preparing frame
[INFO] Configure radio for TX
[INFO] Configure radio for TX
[INFO] Rx Window 1
[INFO] RxDone 12 bytes RSSI: -66 dB SNR: 77 cB
[INFO] Packet for 00000003
[INFO] Packet Received : Port: 0 FCnt: 0000046e Size: 0 ACK: 1 DUP: 0
[INFO] Packet RSSI: -66 dB SNR: 77 cB
[INFO] successfully sent data to gateway
[INFO] Time to next TX: 675

Error:

[INFO] Configure radio for TX
[INFO] Time-on-air for 13 byte: 164
[INFO] Preparing frame
Operator new[] out of memory

[Peter Ferland]

Can you share the code in your main loop where you’re creating and populating the vector?

[Mark Makarychev]

std::vector<uint8_t> payload;

int main() {
    /*--------LoRa Configuration-----------------*/
    // Custom event handler for automatically displaying RX data
    RadioEvent events;
    mts::MTSLog::setLogLevel(mts::MTSLog::TRACE_LEVEL);
    dot = mDot::getInstance();
    logInfo("mbed-os library version: %d", MBED_LIBRARY_VERSION);
    
    // start from a well-known state
    logInfo("defaulting Dot configuration");
    dot->resetConfig();
    dot->resetNetworkSession();

    // make sure library logging is turned on
    dot->setLogLevel(mts::MTSLog::INFO_LEVEL);

    // attach the custom events handler
    dot->setEvents(&events);

    // update configuration if necessary
    if (dot->getJoinMode() != mDot::OTA) {
        logInfo("changing network join mode to OTA");
        if (dot->setJoinMode(mDot::OTA) != mDot::MDOT_OK)
            logError("failed to set network join mode to OTA");
    }
    // in OTA and AUTO_OTA join modes, the credentials can be passed to the library as a name and passphrase or an ID and KEY
    update_ota_config_name_phrase(network_name, network_passphrase, 0, public_network, ack);
    
    // configure the Dot for class C operation
    // the Dot must also be configured on the gateway for class C
    // use the lora-query application to do this on a Conduit: http://www.multitech.net/developer/software/lora/lora-network-server/
    logInfo("changing network mode to class C");
    if (dot->setClass("C") != mDot::MDOT_OK)
        logError("failed to set network mode to class C");
    
    if (dot->setTxDataRate(DataRate) != mDot::MDOT_OK)
        logError("failed to set TX DR");
        
    if (dot->setTxPower(20) != mDot::MDOT_OK)
        logError("failed to set TX Power");
        
    // save changes to configuration
    logInfo("saving configuration");
    if (!dot->saveConfig())
        logError("failed to save configuration");

    // display configuration
    display_config();
    
    
    //Tell the gateway that this device is up and running
    if (!dot->getNetworkJoinStatus())
        join_network();
    payload.push_back((int8_t) var0);
    send_data(payload);

/*--------------LOOP------------------------------------*/
    while (true) {  
        if (refresh_Timer.read_ms() >= refresh_Time) {
                refresh_Timer.reset();
                // join network if not joined
                if (!dot->getNetworkJoinStatus())
                    join_network();
                //std::vector<uint8_t> payload; //tried changing the payload variable's 
	       //scope so that it would always be destroyed after sending, but that didn't help
                payload.clear();
                //std::vector<uint8_t>(payload).swap(payload); // also didn't help
                payload.push_back((int8_t) var1);
                payload.push_back((int8_t) var2);
                payload.push_back((int8_t) var3);
	        //var4 and var5 are uint32_t (conversion from float)
                for(int i=24; i>=0; i-=8)
                    payload.push_back((var4>>i)&0xFF);
                for(int i=24; i>=0; i-=8)
                    payload.push_back((var5>>i)&0xFF);
                payload.push_back((int8_t) var6);
                payload.push_back((int8_t) var7);
                logInfo("Time-on-air for %d byte: %ld",payload.size(), dot->getTimeOnAir(payload.size()));
                send_data(payload);
                next_tx = dot->getNextTxMs();
                logInfo("Time to next TX: %ld", next_tx);
        }
        
        //soft reset solution I was talking about
        if (reset_cpu_Timer.read_ms() >= reset_cpu_Time)
            dot->resetCpu();
    }
    return 0;
}

[Mark Makarychev]

Forgot this little bit:

void send_data(const std::vector<uint8_t>& data) {
    uint32_t ret = dot->send(data);
    if (ret != mDot::MDOT_OK) {
        logError("failed to send data to %s [%d][%s]", dot->getJoinMode() == mDot::PEER_TO_PEER ? "peer" : "gateway", ret, mDot::getReturnCodeString(ret).c_str());
    } else {
        logInfo("successfully sent data to %s", dot->getJoinMode() == mDot::PEER_TO_PEER ? "peer" : "gateway");
    }
}

[Peter Ferland]

Can you try using libmdot-dev-mbed5?

• This reply was modified 7 years, 2 months ago by Peter Ferland.

[Mark Makarychev]

I was originally using libmdot-dev-mbed5 and moved to the “not-dev” version, thinking that this might be an instability in the “dev” version. Although, I’m not sure I was using the mbed-OS version the libmdot-dev-mbed5 was tested against. I’ll try one more “clean” test tomorrow. By the way, just to be clear: is it that you can’t recreate the error, or has the library just not been tested in such a manner before?

[Peter Ferland]

They have been tested in similar conditions without failure. I have seen memory leaks in similar conditions in customer code but we have not been able to reproduce a problem with the library on our own systems. Any extra info would be helpful.

• This reply was modified 7 years, 2 months ago by Peter Ferland.

[Mark Makarychev]

Using mdot-library revision 2.0.16-7-ga61aab1 with mbed-os-5.3.4 I got to 1600+ frames (FCnt over 640) without any errors. Not bad so far! However, these 1600+ frames were sent with duty-cycle restrictions turned off to speed up the test. I’ll write back once more after doing a true test with the duty-cycle enabled.

[Mark Makarychev]

Still getting the error after around one hour of sending with the duty-cycle enabled. I’m not dismissing the idea that it might be some other library’s fault, but it’s just strange, that the out of memory error always occurs on the “preparing frame” stage, and not when other libraries’ functions are being called. I’ll do some more testing and try to get an error from other parts of my code – if something is leaking every hour with sends, it should still leak without them, right? Just might take longer. I’ll write back if I find anything.

[Peter Ferland]

We ran a similar example (modification of the loop in ota_example) overnight in a constant transmit/receive loop with a Conduit without going to sleep. Its still running after 45k packets.

The only other vector related thing that comes to mind to check is using a resize before the loop and then using array access instead of push_back. I’ll try one more experiment with running exactly your code and run that over the day.

[Mark Makarychev]

Sorry for the false alarm, it wasn’t a leak in your library after all! Hurray!!! Found a very helpful page: https://docs.mbed.com/docs/mbed-os-handbook/en/5.2/advanced/runtime_mem_trace/, turned on memory tracing for mbed OS (how very thoughtful of them!), and found that a poorly written I2C library was not freeing memory after allocating it =/. Thanks for your help anyways!

[Tom Hill]

Hi Mark,

Are you using mbed os5 I2C api’s in mdot, and was that causing the memory leak? If so can you share some details regarding that? As I am planning to use the I2C as well, where the MDot behaves as the I2C Slave.

Thanks,
Yogesh

[Mark Makarychev]

Hello Yogesh,

No, I just found some library using ARM mbed’s built in search functionality. I wasn’t aware os5 had an I2C API. Might even check it out myself now, haha!

[Author]

Posts