GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Heap unification by JF002 · Pull Request #1709 · InfiniTimeOrg/InfiniTime

Conversation

@JF002

@JF002 JF002 commented Mar 26, 2023

Copy link
Copy Markdown
Collaborator

This PR unifies the 3 memory heaps used in InfiniTime (stdlib, LVGL and FreeRTOS) into a single memory space managed by FreeRTOS.

The goals are

  • Reduce the memory overhead caused by those 3 heaps
  • Provide an easy way to monitor the memory usage at runtime
  • Provide a better overview of the memory allocation.

According to the memory analysis I've previously done, dynamic memory allocations (which consume heap memory) are done in multiple parts of the code:

  • LVGL
  • use of new() and delete() (std::string, std::unique_ptr, apps loading in InfiniTime)
  • LittleFS
  • NimBLE

LVGL support custom memory manager, so I simply edited the configuration to point the alloc/free functions to FreeRTOS memory manager.
For other parts of the code that use malloc(), delete(), new(), free(), I defined custom implementations of malloc(), free(), realloc() so that they'll be linked in the code in place of the ones from stdlib. This ensures that the heap from stdlib is not used anymore.

In the end, here's a rough overview of the memory allocation:

  • Total memory available : 65536B
  • Memory allocated : 63728B + 1024B (stack) = 64752B
  • FreeRTOS heap : 40960B

This leaves 784B unallocated at build time.

I used this branch for multiple days and the minimum memory available in the FreeRTOS heap was around 6KB. Which means that we still have ~6KB of RAM available for static allocation (we'll need a bit of this for the new HR implementation) or for dynamic allocation (for new apps, fonts,...).

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Riksu9000, minacode, FintasticMan, and Tiggilyboo reacted with thumbs up emoji
All reactions
  • 👍 4 reactions
JF002 added 4 commits March 26, 2023 14:21
@JF002
Copy the memory manager heap_4.c from the FreeRTOS distribution to th…
561ee75
…e InfiniTime project codebase. This will allow to implement custom functionalities that are not available in the original implementation.
@JF002
Add pvPortRealloc() in heap_4_infinitime.c. An implementation of real…
2edd2f8
…loc() is needed by NimBLE.
@JF002
Unify all heaps (stdlib + LVGL + FreeRTOS) into a single heap managed…
36b69de
… by FreeRTOS and heap_4_infinitime.c.

LVGL supports custom implementation of malloc() and free() so using pvPortMalloc() and vPortFree() is just a matter of setting the right variables.

Other libraries (NimBLE, LittleFS) and InfiniTime code (new) call malloc() and free() from stdlib. InfiniTime now provides the file stdlib.c that provides a custom implementation for malloc(), free(), calloc() and realloc(). This ensures that all calls to the standard allocator are redirected to the FreeRTOS memory manager.

Note that realloc() is needed by NimBLE.
@JF002
Enable malloc error and stack overflow error detection in FreeRTOS. C…
87e8c9b
…ount them and display them in the SystemInfo app.
@github-actions

github-actions Bot commented Mar 26, 2023 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

Build size and comparison to main:

Section Size Difference
text 406104B -352B
data 940B 0B
bss 62772B 9212B
All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Riksu9000
Merge branch 'main' into heap-unification
98dc772
Riksu9000
Riksu9000 reviewed Mar 26, 2023
View reviewed changes

@Riksu9000 Riksu9000 left a comment

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a great improvement. I have just a few comments.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread src/main.cpp
Comment on lines +151 to +152
int mallocFailedCount = 0;
int stackOverflowCount = 0;

@Riksu9000 Riksu9000 Mar 26, 2023

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You shouldn't be able to write to these variables from anywhere, but I'm not sure where else they could be placed.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@JF002 JF002 Apr 1, 2023

Copy link
Copy Markdown
Collaborator Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest, I do not really like those global variables either. They are not strictly needed at runtime, but they provide useful information at development time (and also in the SystemInfo app).

Those variables could be wrapped into a object similar to other Components and passed to other modules by ref. I felt it was a bit over-engineered for just 2 variables, but it could be used for other purposes in the future. What do you think?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@Riksu9000 Riksu9000 Apr 3, 2023

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess they could be moved into an anonymous namespace and then add new functions to just get the value. In fact a lot of the global objects are probably intended to be in an anonymous namespace.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@JF002 JF002 Apr 8, 2023

Copy link
Copy Markdown
Collaborator Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're probably right about the global objects that should be in an anonymous namespace.
However, I don't think it's possible specifically for those 2 variables : that are used in an extern "C" scope (in main.cpp), and access using an extern declaration in SystemInfo.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread src/FreeRTOSConfig.h
#define configMAX_PRIORITIES (3)
#define configMINIMAL_STACK_SIZE (120)
#define configTOTAL_HEAP_SIZE (1024 * 17)
#define configTOTAL_HEAP_SIZE (1024 * 40)

@Riksu9000 Riksu9000 Mar 26, 2023

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That unallocated memory is essentially wasted right? So can we make the heap automatically use all available space?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@JF002 JF002 Apr 1, 2023

Copy link
Copy Markdown
Collaborator Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Memory region         Used Size  Region Size  %age Used
           FLASH:      406164 B     474632 B     85.57%
     SPARE_SPACE:          0 GB        12 KB      0.00%
             RAM:       63720 B        64 KB     97.23%

In this case, 63720B + 1024B (for the main stack) are allocated, which leaves 792B un-allocated space which are basically wasted (not used by the application).

So yes, that would be really nice if we could make the heap automatically use all the available space. Not sure how to do this tough...

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
FintasticMan
FintasticMan reviewed Mar 26, 2023
View reviewed changes

@FintasticMan FintasticMan left a comment

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like these changes, it will make monitoring heap usage easier and lower memory overhead.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
Comment thread src/FreeRTOS/heap_4_infinitime.c Resolved

Uh oh!

There was an error while loading. Please reload this page.

@JF002 JF002 mentioned this pull request Mar 26, 2023
FLASH and RAM memory analysis #313
Closed
@kieranc

kieranc commented Mar 30, 2023

Copy link
Copy Markdown
Contributor

Initital testing looks good, I can switch from PTS to G7710 to Infineat without corrupted fonts.

JF002 reacted with thumbs up emoji
All reactions
  • 👍 1 reaction

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@JF002 JF002 added this to the 1.13.0 milestone Apr 2, 2023
JF002 added 2 commits April 8, 2023 15:30
@JF002
Refactor pvPortRealloc() to improve readability.
e278a58
@JF002
Fix code formatting
ec1eef8
FintasticMan
FintasticMan reviewed Apr 9, 2023
View reviewed changes
Comment thread src/FreeRTOS/heap_4_infinitime.c
puc -= xHeapStructSize;

// This casting is to keep the compiler from issuing warnings.
pxLink = (void*) puc;

@FintasticMan FintasticMan Apr 8, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why cast to a void pointer and not a BlockLink_t pointer?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@JF002 JF002 Apr 9, 2023

Copy link
Copy Markdown
Collaborator Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question... I took my inspiration from the original code here

InfiniTime/src/FreeRTOS/heap_4_infinitime.c

Line 287 in e278a58

pxLink = ( void * ) puc;
.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

@FintasticMan FintasticMan Apr 10, 2023

Copy link
Copy Markdown
Member

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have tested it with (BlockLink_t*), and it doesn't give any warnings. As far as I'm concerned, it doesn't really matter either way, I just think that casting to a BlockLink_t pointer is slightly more clear.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
JF002 added a commit to InfiniTimeOrg/InfiniSim that referenced this pull request Apr 9, 2023
@JF002
Integrate the new heap implementation from InfiniTime (InfiniTimeOrg/…
f735bb9
…InfiniTime#1709).

Since FreeRTOS.h, portmacro_cmsis.h and task.h are now built in C (by lv_mem.c), I had to change some includes and declarations to make them compatible with a C compiler.

Integrating the new memory management from InfiniTime in InfiniSim is not easy because InfiniSim does not include the whole FreeRTOS. Which means that, for example, pvPortMalloc() and vPortFree() are not accessible from InfiniSim.
As a first step, I provided custom implementations for pvPortMalloc(), vPortFree() which are based on ... malloc(). These function keep track of the memory that is currently allocated so that xPortGetFreeHeapSize(), xPortGetMinimumEverFreeHeapSize() return something.

Not that this implementation do not keep track of all the memory allocations done in InfiniTime. It can only "see" those done via pvPortMalloc(). It means that the available memory displayed by InfiniSim will probably be very optimistic.
@JF002 JF002 mentioned this pull request Apr 9, 2023
Integrate the new heap implementation from InfiniTime InfiniTimeOrg/InfiniSim#97
Merged
FintasticMan
FintasticMan approved these changes Apr 10, 2023
View reviewed changes
Riksu9000
Riksu9000 approved these changes Apr 11, 2023
View reviewed changes
@JF002 JF002 mentioned this pull request Apr 16, 2023
1.11 - Chimes setting makes the font in the G7710 Watchface disappear #1411
Closed
1 task done
NeroBurner pushed a commit to InfiniTimeOrg/InfiniSim that referenced this pull request May 1, 2023
@JF002 @NeroBurner
Integrate the new heap implementation from InfiniTime (InfiniTimeOrg/…
4e6a2b7
…InfiniTime#1709).

Since FreeRTOS.h, portmacro_cmsis.h and task.h are now built in C (by lv_mem.c), I had to change some includes and declarations to make them compatible with a C compiler.

Integrating the new memory management from InfiniTime in InfiniSim is not easy because InfiniSim does not include the whole FreeRTOS. Which means that, for example, pvPortMalloc() and vPortFree() are not accessible from InfiniSim.
As a first step, I provided custom implementations for pvPortMalloc(), vPortFree() which are based on ... malloc(). These function keep track of the memory that is currently allocated so that xPortGetFreeHeapSize(), xPortGetMinimumEverFreeHeapSize() return something.

Not that this implementation do not keep track of all the memory allocations done in InfiniTime. It can only "see" those done via pvPortMalloc(). It means that the available memory displayed by InfiniSim will probably be very optimistic.
NeroBurner pushed a commit to InfiniTimeOrg/InfiniSim that referenced this pull request May 3, 2023
@JF002 @NeroBurner
Integrate the new heap implementation from InfiniTime (InfiniTimeOrg/…
8ae5ba7
…InfiniTime#1709).

Since FreeRTOS.h, portmacro_cmsis.h and task.h are now built in C (by lv_mem.c), I had to change some includes and declarations to make them compatible with a C compiler.

Integrating the new memory management from InfiniTime in InfiniSim is not easy because InfiniSim does not include the whole FreeRTOS. Which means that, for example, pvPortMalloc() and vPortFree() are not accessible from InfiniSim.
As a first step, I provided custom implementations for pvPortMalloc(), vPortFree() which are based on ... malloc(). These function keep track of the memory that is currently allocated so that xPortGetFreeHeapSize(), xPortGetMinimumEverFreeHeapSize() return something.

Not that this implementation do not keep track of all the memory allocations done in InfiniTime. It can only "see" those done via pvPortMalloc(). It means that the available memory displayed by InfiniSim will probably be very optimistic.
mark9064 added a commit to mark9064/InfiniTime that referenced this pull request May 8, 2023
@mark9064
Heap unification InfiniTimeOrg#1709
a7b9dbd
commit ec1eef8
Author: Jean-François Milants <[email protected]>
Date:   Sun Apr 9 10:46:55 2023 +0200

    Fix code formatting

commit e278a58
Author: Jean-François Milants <[email protected]>
Date:   Sat Apr 8 15:30:14 2023 +0200

    Refactor pvPortRealloc() to improve readability.

commit 98dc772
Merge: 87e8c9b 0a4f015
Author: Riku Isokoski <[email protected]>
Date:   Sun Mar 26 20:00:48 2023 +0300

    Merge branch 'main' into heap-unification

commit 87e8c9b
Author: Jean-François Milants <[email protected]>
Date:   Sun Mar 26 14:53:34 2023 +0200

    Enable malloc error and stack overflow error detection in FreeRTOS. Count them and display them in the SystemInfo app.

commit 36b69de
Author: Jean-François Milants <[email protected]>
Date:   Sun Mar 26 14:49:03 2023 +0200

    Unify all heaps (stdlib + LVGL + FreeRTOS) into a single heap managed by FreeRTOS and heap_4_infinitime.c.

    LVGL supports custom implementation of malloc() and free() so using pvPortMalloc() and vPortFree() is just a matter of setting the right variables.

    Other libraries (NimBLE, LittleFS) and InfiniTime code (new) call malloc() and free() from stdlib. InfiniTime now provides the file stdlib.c that provides a custom implementation for malloc(), free(), calloc() and realloc(). This ensures that all calls to the standard allocator are redirected to the FreeRTOS memory manager.

    Note that realloc() is needed by NimBLE.

commit 2edd2f8
Author: Jean-François Milants <[email protected]>
Date:   Sun Mar 26 14:31:59 2023 +0200

    Add pvPortRealloc() in heap_4_infinitime.c. An implementation of realloc() is needed by NimBLE.

commit 561ee75
Author: Jean-François Milants <[email protected]>
Date:   Sun Mar 26 14:21:35 2023 +0200

    Copy the memory manager heap_4.c from the FreeRTOS distribution to the InfiniTime project codebase. This will allow to implement custom functionalities that are not available in the original implementation.
@JF002
JF002 merged commit 8abfd25 into main May 18, 2023
@FintasticMan
FintasticMan deleted the heap-unification branch August 3, 2023 11:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@FintasticMan FintasticMan FintasticMan approved these changes
+1 more reviewer
@Riksu9000 Riksu9000 Riksu9000 approved these changes
Reviewers whose approvals may not affect merge requirements

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

1.13.0

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

4 participants

@JF002 @kieranc @Riksu9000 @FintasticMan
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.