Rockbox Technical Forums

Support and General Use => Audio Playback, Database and Playlists => Topic started by: Carlo on February 12, 2023, 01:23:08 PM

Title: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 12, 2023, 01:23:08 PM
After updating Rockbox on my Sansa Clip+ to the 20230208 daily build I'm frequently experiencing "Data abort" errors when switching tracks or fast forwarding. They started to happen right after updating from a last december build. A sample of the error summary:

Data abort at 30063908
FSR 0x8
(domain 0, fault 8)
address 0x57ee557a
pc:30063908
sp:300d47a8
bt end

The MP3 files I'm listening to have been transferred on the Clip+ several months ago, so no recent additions, and the error manifest itself on average after 15 minutes, sometimes just after two or three minutes from booting. It cannot be reproduced, but it's triggered by switching tracks and by fast forward/reverse. Daily build from today shows the same behavior.

I've performed a r/w test on the internal memory and there are no reported issues.

Any suggestion about how to further diagnose and solve the issue? Thanks.

Edit: it seems it could be related to this bug report, https://www.rockbox.org/tracker/task/13388 . I tried to register on the bug tracker but I receive the error "Fatal error: Array and string offset access syntax with curly braces is no longer supported in /home/rockbox/flyspray/vendor/dapphp/securimage/securimage.php on line 2209".

Edit 2: it happens with the build from 20230113 too, so the bug was introduced earlier.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on February 13, 2023, 12:20:58 AM
I have a clip+ as well and I can't repro this so far

if you want to give me the dataabort on this build we can try to trace it
I've included the .map file which should show us where the program counter is pointing

https://www.mediafire.com/file/sxd4vdznpuj8f0c/ClipPlus_3bb75e0039-230213_rockbox-full.zip/file

be sure to rename your old .rockbox folder and do a clean 'install'

it doesn't mean there is or isn't an issue with your sd card though but i'd try a different one anyway
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 13, 2023, 05:56:24 AM
It's unlikely that's an hardware issue, as I've compiled a build from 20221230 and it seems fine so far. I'll give your build a try and report back.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 13, 2023, 01:15:38 PM
The error showed up again when using your build, this is the data:

Data abort at 30062974
FSR 0x8
(domain 0, fault 8 )
address 0xfffffef4
pc:30062974
sp:300d4740

It happened right after booting and switching no more than 6 or 7 tracks. A second one, after a few minutes:

Data abort at 30062318
FSR 0x8
(domain 0, fault 8 )
address 0xac5fdf74
pc:30062318
sp:300d4740
bt end

It seems the only common address between the last two crashes is the stack pointer one. A third one, also happened while cycling through the same directory as the other two crashes:

Data abort at 30063908
FSR 0x8
(domain 0, fault 8 )
address: 0x55555561
pc:30063908
sp:300d47a8
bt end
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on February 13, 2023, 04:11:29 PM
The first two are in the font system and the third appears to be in LCD init_viewport are you using any themes or is it a bone stock install? Also do the songs it crashes on crash everytime or just randomly? any special chars in those filenames ETC?
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 13, 2023, 05:13:34 PM
1) I'm using the Informative Beauty theme, as I've done for years on all my Clip+ players without any issue.

2) The crashes are random, I could cycle 5-6 times through the files in the same directory before getting one.

3) No special characters. Again, last time I added new files on this device was a few months ago, and it's my daily driver - so I noticed the crashes immediately after updating, as until that moment it was working perfectly.

I can confirm beyond any reasonable doubt that the 20221230 build is NOT affected by the issue, while 20230113 is affected.

Thank you for your help!
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on February 13, 2023, 05:16:28 PM
Try to repro the issue using the default theme cabbie the closer you can get me to the issue the less time it'll take to fix it if i can reproduce it it'll be even faster otherwise we will have to bisect through 10-100 builds one at a time
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 13, 2023, 05:27:21 PM
Could the bug be related to the "[BUGFix] ClipPlus move picker below yellow screen split" from 20230113? It wouldn't be that much of a hassle for me to bisect the two weeks between 20221230 and 20230113 until I can pinpoint a bug free build.

I've prepared a Debian vm with the proper compiler toolchain last night, and it doesn't take that long to checkout a specific commit, build and test.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on February 13, 2023, 05:36:45 PM
not unless you are using the keyboard in that run..

git bisect is your next course of action if you can compile builds or I can for you given about 30 mins to an hour
see us on IRC in that case

Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 13, 2023, 05:42:09 PM
I'm able and willing to compile builds, will start right now with 20230106 and see how it goes.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on February 13, 2023, 05:46:54 PM
https://gerrit.rockbox.org/r/c/rockbox/+/5112
parent 78718aa7eb < here is where it works is my guess
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 13, 2023, 05:58:24 PM
It seems that change was committed on Jan 29, but the crashes already happen with the daily Jan 13 build.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on February 15, 2023, 01:22:36 AM
I think I found it its been merged so try the latest daily
https://gerrit.rockbox.org/r/c/rockbox/+/5123
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 15, 2023, 05:52:00 AM
Thank you very much, I will try the latest daily for a couple of hours later today and report back.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 15, 2023, 12:13:30 PM
Sadly I have bad news: I had an hard freeze with the Informative Beauty theme a couple of minutes after booting, but when I switched to the default Cabbiev2 theme it worked fine for more than two hours. I'll do more testing tonight.

Edit: as puzzling as it sounds, after performing a clean install and manually changing all the relevant settings I didn't have any crashes for more than one hour. 
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 16, 2023, 10:02:44 AM
I'm getting more and more data aborts.. transcript of the last one:

Data abort at 30063908
FSR 0x8
(domain 0, fault 8 )
address 0x772046d0
pc: 30063908
sp: 300d47a8
bt end

I'm running the 20230215 daily with the following config (recreated from scratch and not imported from a previous firmware version):

volume: -40
bass: 22
treble: 1
repeat: all
contrast: 50
battery display: numeric
eq enabled: on
eq low shelf filter: 32, 7, -1
surround enabled: 8
afr enabled: moderate
pbe: 75
compressor knee: hard knee
fms: -
wps: /.rockbox/wps/informativebeauty.wps
iconset: -
viewers iconset: -
qs top: -
qs left: shuffle
qs right: repeat
qs bottom: -
Semitone pitch change: on

I do not have installed the additional fonts package, just the Informative Beauty theme. Tonight I'll try to perform more bisecting and try to reproduce the bug with the steps bahus pointed out.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 17, 2023, 12:42:24 PM
(Replying here from the Sansa Zip thread)

I'm still getting crashes, albeit it seems they're not as frequent as before - had three in a hour and half. Could you please provide me with your build+mapfile? Thanks.

If it may be of any help, I've noticed the crashes do happen more often while cycling through relatively few files (10-15) inside the same directory, instead of larger (100+ files) ones, and they always happen immediately after switching to the next or previous track. The earliest build I can confirm has the bug is this one:

2023-01-15 buflib: Add CONFIG_BUFLIB_BACKEND for selecting a buflib backend

Could the bug be related to one of those commits?

2023-01-12 add chunk_alloc to playlist.c #2   or
2023-01-12 [BugFix] playlist.c DIRCACHE stop scanning when changing indices
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on February 17, 2023, 03:13:11 PM
Who knows random bugs are extremely hard to figure out without a way to reliably reproduce them
if you can get me something that will crash everytime git bisect is the fastest way to figure it out..

Speculating and guessing at what commit is a hard way to find bugs
unless its a glaringly obvious bug

The last two bugs I fixed were not obvious to me at first glance
But I had an order of operations that made for a reproducible bug
then I had a place to start looking and a way to check if it was fixed after.

If they were obvious I'd have caught them when I was coding and testing
the same goes for the others commits as well and probably more so..

if you are compiling your own builds you can grab the mapfile for your build its going to be in the build dir and its called rockbox.map
every build updates the mapfile so you just need to type
Code: [Select]
checkout master
Code: [Select]
make -j2 && make fullzipwait for rockbox to build and open 'rockbox.map' in the same directory as 'rockbox-full.zip'

Later, if you want to show us just copy and paste the section your pc was pointing at with a bit of context

example:
so I got a pc of 3006a080
closest I can find is 0x00000000300699b4                font_load_ex

Quote
.text        0x0000000030069180     0x1098 /rockbox/.BUILDS/ClipZip/firmware/libfirmware.a(font.o)
                ..........
                0x00000000300697a0                font_get
                0x0000000030069800                font_get_width
                0x00000000300698bc                font_get_bits
                0x00000000300699b4                font_load_ex
                0x000000003006a128                font_load
                0x000000003006a138                font_getstringnsize
                .........

so now i'll starting looking for commits that touched here first
but the thing is even armed with that data it still might be the wrong place
who's to say its not code elsewhere, a buffer overrun could cause issues
in a different place too by overwriting someone else's memory or code

your map file could help with that too once you armed with enough info to go looking for memory addresses of buffers and functions..

still for recently introduced bugs git bisect is way faster than all of that it allows you to narrow it down to what commit broke it
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 17, 2023, 04:32:45 PM
I am indeed using bisect too, however as you said there's no surefire way to trigger the bug, so I may tag as good a commit that's actually a bad one. After I updated to the latest master and installed the build on my Clip+ the bug showed up almost 40 minutes after booting, so it may indeed take quite a bit of time to trigger it. Again, I have used Rockbox for years on several devices with a very similar configuration and setup, and I never encountered even a single data abort error.

The issue might also be present in the 20221230 and 20221231 builds I've used for several hours without encountering any data abort.

I'll keep bisecting and trying to pinpoint a specific commit, however I'm taking things slow as I want to be absoutely sure that a given commit is good before proceding onto the next one. I'll try testing on multiple Clip+ at once.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 18, 2023, 10:14:50 AM
A quick update about the bisecting saga: I've used the following build for more than one hour and didn't get any crash:

2023-01-12 Add INIT_ATTR to system_init()    <- This one seems good

After the next bisection step I compiled and installed the following:

2023-01-13 toolchain: Bump zlib to 1.2.13 due to 1.2.12 being withdrawn   <- Got a crash here

And got a crash before the two minutes mark:

Data abort at 30062e54
FSR 0x8
(domain 0, fault 8 )
address 0x8831881d
pc: 30062e54
sp: 300d3960
bt end

Looking at the rockbox.map it seems 0x30062e54 is related to font_load_ex() as you already discovered:

0x0000000030061fdc                font_get_bits
0x00000000300620d4                font_load_ex
0x0000000030062850                font_load

The only commit between 20230112 and 20230113 related to fonts it's this one:

"Avoid using buflib names for storing font paths"
https://git.rockbox.org/cgit/rockbox.git/commit/?id=879888b158376f1ea2c92dd49e0c7617d07fd5b2

I'll perform some more testing later tonight.


Edit: 20130113 "Remove buflib allocation names, part one" crashes too. I've narrowed down the possible list to just 7 commits or so. The latest crash:

Data abort at 30063348
..
pc:30063348
sp:300d4040

0x00000000300632f0                lcd_setfont
0x0000000030063304                lcd_getfont
0x0000000030063318                lcd_getstringsize


Edit 2: this is my current bisection range:

2023-01-13 Remove buflib allocation names, part one  <- Bad (crash < 2 minutes after booting)
2023-01-13 Avoid using buflib names for storing font paths  <- Bad (crash < 2 minutes after booting)
2023-01-13 keyboard.c make editline respect current UI font <- Bad (crash after 30-40 mins, pc 3006241c, font_get)
2023-01-12 add chunk_alloc to playlist.c #2
2023-01-12 [BugFix] playlist.c DIRCACHE stop scanning when changing indices  <- Good (no crash after 1+ hour)
2023-01-12 Fix red in 7e5fc4076a
2023-01-12 Add INIT_ATTR to i2c_init()
2023-01-12 Add INIT_ATTR to system_init() <- Good (no crash after 1h+ hour)

The pc on my latest crash (2023-01-13 keyboard.c make editline respect current UI font) is 3006241c:

0x00000000300623f8   font_get
0x0000000030062458  font_get_width
0x0000000030062514  font_get_bits

However I have the strong impression that the latest crash may be caused by the font_filename or chunk_alloc() bugs that you've fixed in your last commits, as it happened after at least 30 minutes from booting, instead of after just a few minutes like with the later builds. The issue that still hasn't been fixed gets triggered in just a few minutes on my device.

I will test "2023-01-13 Avoid using buflib names for storing font paths" and see how long does it take to crash, as I believe the latest crash isn't related to the issue that remains undiscovered. (It crashes after a few minutes).


Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 18, 2023, 01:59:59 PM
Creating a new message to avoid confusion. This is my final bisecting report:

2023-01-13 Remove buflib allocation names, part one  <- Bad (crash < 2 minutes after booting)
2023-01-13 Avoid using buflib names for storing font paths  <- Bad (crash < 2 minutes after booting)
2023-01-13 keyboard.c make editline respect current UI font <- Bad (crash after 30-40 mins, pc 3006241c, font_get, bug already fixed?)
2023-01-12 add chunk_alloc to playlist.c #2
2023-01-12 [BugFix] playlist.c DIRCACHE stop scanning when changing indices  <- Good (no crash after 3+ hours)
2023-01-12 Fix red in 7e5fc4076a
2023-01-12 Add INIT_ATTR to i2c_init()
2023-01-12 Add INIT_ATTR to system_init() <- Good (no crash after 1+ hour)

I believe the bug that has yet to be identified and fixed has been introduced in "2023-01-13 Avoid using buflib names for storing font paths" and the crash I had with "2023-01-13 keyboard.c make editline respect current UI font" was related instead to one of the bugs you've fixed in the latest commits.

"2023-01-13 Avoid using buflib names for storing font paths" crashes extremely quickly for me, I just need to skip a few songs inside a directory to trigger the bug. I believe that's the culprit.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on February 18, 2023, 10:23:39 PM
is this one good or bad '2023-01-12 add chunk_alloc to playlist.c #2'
I would figure its the culprit..
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 19, 2023, 07:08:37 AM
I thought that I already had implicitly tested "2023-01-12 add chunk_alloc to playlist.c #2" when I tried "2023-01-13 keyboard.c make editline..", as it's the next immediate commit and the keyboard changes seem absolutely minimal and safe.

"2023-01-13 Avoid using buflib names.." crashes almost immediately. This is the first commit that's extremely unstable for me. I simply have to boot the Clip+, start playing a file at random and then quickly skipping tracks to get a crash. The crash is triggered immediately after a song is loaded, never while it's playing.

"2023-01-13 keyboard.c make editline.." (so, transitively, "2023-01-12 add chunk_alloc.." too) crashed once after quite a bit of time, so I believe the crash itself may be caused by one of the chunk_alloc bugs you've already fixed, as the different bug I'm experiencing is triggered way more quickly.

"2023-01-12 [BugFix] playlist.c DIRCACHE stop scanning" was tested extensively and never crashed.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on February 19, 2023, 11:32:00 AM
the thing is "2023-01-13 Avoid using buflib names.."
is probably a red herring likely its just more susceptible
to the bug since a buffer overrun might be more likely to wipe out important data
lets try removing that chunked alloc and i'll give you a build
to try
I'd expect there to be  a bug in my code more than amachronics anyway
but it still might be either and moving stuff around or adding more memory pressure is
causing an existing bug to be exposed

here is a build to try with playlist name chunking disabled
since you are building your own builds you should just be able to compile it
Code: [Select]
git checkout 2703cc0599034f6d72b2c0eeee7bd4d6ac399bed

https://gerrit.rockbox.org/r/c/rockbox/+/5147
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 19, 2023, 12:03:16 PM
Thank you, I'm going to compile a new build right now.

In order to fetch your changes I had to do the following:

rb@debian:~/rockbox$ git ls-remote | grep -i '2703cc05'
From git://git.rockbox.org/rockbox
2703cc0599034f6d72b2c0eeee7bd4d6ac399bed        refs/changes/47/5147/1

rb@debian:~/rockbox$ git pull origin refs/changes/47/5147/1

I can confirm playlist.c and playlist.h have been updated, and I'll test the new build and give you an update ASAP.

Edit: no crashes after one hour of testing.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 19, 2023, 05:07:26 PM
I've used my Clip+ for more than two hours without a single issue, so it seems your workaround nailed it. Thank you!
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on February 19, 2023, 09:47:54 PM
we;; its apparently something with my code do me a favor and continue testing that build
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 20, 2023, 05:12:30 AM
Absolutely, I will continue to use it on my daily driver. I'm also available for testing your addional builds on my other Clip+ players before they get pushed to master.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 20, 2023, 12:35:53 PM
I've used my Clip+ for several hours and it's been completely bug free. Zero crashes!
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on February 21, 2023, 02:25:59 PM
IVE added in a test function to test the data integrity of the chunk allocator
But IVE yet to find any bugs in my implementation
Ill keep trying but it makes me wonder if its just putting pressure on the buflib causing a bug to appear or if there is something IM missing in the playlist functions using chunk_alloc
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 22, 2023, 05:57:58 AM
I can confirm your modified build has completely solved the issue. Can I provide any further debugging help?
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on February 22, 2023, 11:02:56 AM
can you try these builds?
run this one for a few hours to see if this fixes the issue
https://gerrit.rockbox.org/r/c/rockbox/+/5150
^this was the only other potential bug I found in chunk_alloc and i'm not able to get it to fail in testing
so its just a guess

if that still crashes then please run this one short term
https://gerrit.rockbox.org/r/c/rockbox/+/5149

it should take about 2 minutes to run when you start a track from the file browser

it runs a test allocating random length strings in the chunk_alloc and checking their integrity
hopefully it causes a data abort if not I think we need to look elsewhere
maybe with what playlist.c is doing at the time

Or what is it doing with those strings some overrun??
They just appear to be indexed and copied so I'm still at a loss there


Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 22, 2023, 05:04:08 PM
Sure, I've compiled build 5150 (2f71430) and installed it on my daily driver. I'll keep you updated.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 23, 2023, 09:53:45 AM
Build 5150 (2f71430) is unstable for me. Pc of the latest crash is 30075094, this is my mapfile:

0x0000000030074fd0                font_cache_create
0x0000000030075074                font_cache_get
0x000000003007a160                browse_fonts

I'll compile 5149 later and give it a try.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on February 23, 2023, 10:04:33 AM
it keeps coming back to this font code
I'll try disabling the dynamic name for the font next
I think there is something else going on unless the second patch
up there shows something crashy but I think it might just be uncovering a different bug

Thanks for testing BTW I haven't been able to reproduce any of this on my clip+
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on February 23, 2023, 10:15:26 AM
here is a patch disabling dynamic font names try this one if 5149 doesn't crash
https://gerrit.rockbox.org/r/c/rockbox/+/5151
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 23, 2023, 10:25:00 AM
5149 doesn't crash for me, I've tried several times and the log always says "err:0, tests 500 longest..". I've tried it in various directories of different sizes and the number of errors is always 0.

I'll give 5151 a try shortly.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on February 23, 2023, 10:42:10 AM
5149 basically fills the whole buffer with randomly sized strings and then checks their integrity
if we aren't getting any errors then nothing got corrupted in that test. It also allocates a second buffer at the same time
to try and cause issues.

so I don't think its the chunk_allocator causing the issue I think its just uncovering another bug

but I might be wrong..
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 23, 2023, 11:08:12 AM
5151 (a70c0833..) crashes almost immediately too. Pc 300638c0, mapfile:

30063890 lcd_getstringsize
300638a4 lcd_init_viewport
30063920 lcd_set_viewport_ex

Let me provide a recap:

5147 - Good
5149 - No memory corruption
5150 - Crash after a few minutes
5151 - Crash in less than two minutes

I always triple-check that I'm actually running the intended build by comparing the version string from System -> Rockbox Info
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on February 23, 2023, 12:14:24 PM
Something is definitely up here i dont know what but could you give me a recap on your exact procedure for reproducing this?

Even down to the mundane..
Like i start at the main menu i go to files i choose a file in a directory of 25 files
It goes to the WPS i play a few songs in the WPS
Then i fast forward a few songs and skip back a song or two

I hit the back button it returns to the main menu
Then i go to the file browser and choose a new directory with 21 songs
I play a file and it crashes

OR
If you could and this is a big ask could you get a clean SD card and do an install there with like 10 songs
Or less and get a reproducible set of files based on the current
Dev version that you can get the bug reproduced on
And zip that up and upload it somewhere so i can see if I can reproduce it on my clip+
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 23, 2023, 01:58:19 PM
My exact procedure is extremely simple:

Theme: Informative Beauty

- Turn on the Clip+;
- From the menu I choose "Files" -> "Music";
- Choose a directory at random, containing an average of 15-30 files;
- Start playing a file at random;
- Repeteadly skip to the next track from the WPS.

The crash happens as soon as I have skipped a few tracks. It never ever happened while playing a given track, only right after skipping to the next track. Moreover, the bug happens at random, there's no reliable way to trigger it, as it may happen after I have already skipped several times through the same tracks inside the same directory, e.g.:

- From the directory "dirname" containing 10 files I start playing the first one;
- From the WPS, I start skipping to file 2 -> file 3 -> file 4.. the crash happens here.
OR:
- From the WPS, I start skipping to file 2 -> file 3 -> file 4.. -> file 10, -> back to file 1 -> file 2.. crash.

I always play my files directly from the file browser, as I don't use bookmarks or the database. I'm using the internal Clip+ memory, not a SD card.

I'll try to identify a directory that happens to reliably trigger the bug and provide a zip, but again, the crashes may happen in less than one minute or even after almost one hour of usage.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 25, 2023, 04:15:36 PM
I'm uploading right now an archive that contains my .rockbox directory (build 5151, a70c0833e1) and two MP3 folders. This build is extremely unstable for me, latest crash had a pc of 3006298c:

300628fc  font_enable_all
30062968 font_get
300629c8 font_get_width

I will PM you shortly a GDrive link with the relevant files.

In order to reproduce the bug, simply go inside any one of the two directories with the file browser, start playing a file, then skip tracks, go to the other directory, start playing another file, skip tracks again, etc. This specific build crashes extremely quickly for me.

Edit: the build mapfile is available at https://drive.google.com/file/d/1fbFiezrTogO6yaGOozl8J_xXHOZ0RChn/view?usp=sharing
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on February 26, 2023, 10:27:58 AM
Ive run this build about 3 hours now with your songs and 0 crashes so far

are you able to copy to your other player and crash one of those?

have you tried running from the sd card to eliminate a storage problem?

finally if you can't repro on other players etc we can try upping the CPU voltage..
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 26, 2023, 05:15:39 PM
I can reproduce it by skipping files on a known good SD card and on other players too. See attached screenshot.

Would the internal memory size be a triggering factor?

Edit: I can always quickly reproduce the crash on players with 8 gb of internal memory (with more than 7 gb being used).

Edit 2: the less free space on the player, the quicker the bug seems to be triggered. Skipping tracks as quickly as possible almost always triggers the bug for me.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 27, 2023, 08:08:41 AM
I can reproduce the bug ~80% of the times in less than one minute by doing the following:

- Turn on the Clip+;
- Start playing a random file from the browser as soon as possible;
- Skip forward tracks as quickly as possible from the WPS.

Those four players have 8 GB of internal memory with 7+ GB used and are running build 5151 with the Informative Beauty theme.

PC is either 3006298c or 300638c0.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 27, 2023, 09:04:31 AM
This blue one is a 4 GB Clip+ with 2,5 GB Used.

Prefetch abort at b89ce088
FSR 0x3
(domain 0, fault 3)
pc: b89ce088
sp:300d46c8

Exact same .rockbox folder as the other four. As I wrote in the previous post, just skipping tracks as fast as possible triggers the bug.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on February 27, 2023, 09:42:07 AM
I was finally able to reproduce it by this procedure

start player
goto files
choose new12 (14 files in folder) larger folder seem to be less likely to trigger

press select several times to get it to start the playback over a few times

skip tracks like mad

bam crash
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 27, 2023, 01:08:54 PM
Let's hope fixing the bug will be easier than reproducing it.. I'm available for testing any potential bug fix.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on February 27, 2023, 02:25:02 PM
As long as i can reproduce it i will eventually find it.. Thank you for the help so far
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on February 28, 2023, 12:50:24 AM
the problem (at least the one I'm currently reproducing) appears to have something to do with the scroll engine

I haven't quite got it figured out yet but if I disable scrolling or
in the informativebeauty.wps file disable the last line
Code: [Select]
#%s%al%?It<%Sx(Next:) %It|%?Fn<%Sx(Next:) %Fn|%Sx(End of Song List)>>the crashes cease, so its a clue as to whats happening but I've not got it totally figured out yet
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Frankenpod on February 28, 2023, 02:26:51 AM
Is it definitely the scrolling or could it be the %Sx() translation function?
That used to cause themes to fall over if there was no translation for the given string.
Though it only made the theme fallback to default, not a crash, so perhaps this is something else?#
Does the phrase "End of Song List" definitely have a translated string available?

I wonder if changing the line to

#%s%al%?It<Next %It|%?Fn<Next %Fn|End of Song List>>

...so that it doesn't look for translations, would work?
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on February 28, 2023, 07:04:00 AM
I can confirm that after commenting out the last line of \wps\informativebeauty.wps and reloading the theme the crash isn't reproducible anymore.

Edit: spoke too soon, as Bilgus said I'm getting freezes now instead of data aborts.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on February 28, 2023, 09:40:45 AM
@Frankenpod

I had pushed a patch that made %Sx() fail gracefully a while back
the lang strings do exist

it won't outright dataabort without %Sx() but it will still hang
so might not be the scroll engine but IDK yet
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on March 06, 2023, 01:40:24 PM
IM still working on this...

So far i have it narrowed down to pulling data for the next_track
Its either a bad offset or a buffer overrun or something

Memsetting the whole chunk to 0 stops the crash so something's up
I just DONT know if the playlist chunking is the cause or just bringing exposing an existing bug
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on March 07, 2023, 05:27:40 PM
No worries, take all the time you need. Thank you for your efforts.

Would it make sense to temporarily push playlist.c/.h from 5147 to master until you manage to pinpoint the exact issue? I've been running that build without any issue since you provided it.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on March 09, 2023, 08:17:47 PM
Carlo could you try this build?
https://gerrit.rockbox.org/r/c/rockbox/+/5154
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on March 10, 2023, 05:36:47 AM
Sure, I'm compiling it right now, will give you some feedback ASAP. Thanks.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on March 10, 2023, 06:18:23 PM
Zero crashes in more than three hours of usage, and it seems the bug isn't reproducible at all anymore.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on March 10, 2023, 06:38:52 PM
I still don't know why this works I guess it might become apparent at some point given some more investigation

if you still haven't seen any crashes after a few days can you let me know and I'll push the patch
regardless

so far I've spent a bunch of time on this and clearing the allocated memory is the only thing that works
besides not freeing the buffer which isn't so great after several directories with lots of songs

I'm guessing something is asking for a string way out in the weeds but I've yet to find it...

Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on March 12, 2023, 02:43:38 PM
I can confirm build 5154 solved the issue, even if I'm aware that the real cause of the crashes still hasn't been identified. No crashes after almost ten hours of usage and countless attempts at reproducing the bug on different players.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on March 13, 2023, 05:54:08 PM
I think I've figured out that its not the chunk allocator with this build
could you try it and see if it works for you as well?

https://gerrit.rockbox.org/r/c/rockbox/+/5155

since clearing the ram when we are done with it stops the crash (for me at least)
I think either someone else isn't initializing their allocation or buflib is returning
someone elses data
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on March 14, 2023, 10:42:59 AM
Sure, I'm on it.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on March 16, 2023, 06:00:16 PM
No crashes after several hours of usage and attempts to reproduce the original issue. I believe we're in the clear.
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on March 16, 2023, 09:44:00 PM
I don't know if thats good or bad news

Now I don't know where to start but I'll update this thread when I figure it out
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on March 25, 2023, 11:51:33 AM
Is the "playlist: Remove in-ram filename storage" commit a bugfix for this issue?
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Bilgus on March 25, 2023, 12:37:03 PM
yep I couldn't reproduce after this so do test!
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: bahus on March 26, 2023, 09:19:26 AM
Something is not right with latest playlist changes. With Resume Playback in particular.
Bookmark on Stop option no longer works for me after clicking Resume Playback from main menu. Also after Resume Playback playlist is considered as dynamic as it asks "Erase dynamic playlist?" on attempt to play another file (I suspect that's the main reason why Bookmarking is not working)
Title: Re: Data abort on Sansa Clip+ with latest daily build
Post by: Carlo on April 28, 2023, 05:09:42 PM
I've used the new builds extensively, and the data abort issue was completely resolved for me. I've noticed however that playing a file from a larger directory (500+ songs) for the first time now takes several seconds of "Loading..", instead of being instantanous like before the recent playlist changes, and navigating said directories feels quite sluggish. Is this behaviour related to switching the playlist from RAM to a plain file?

For comparison, the old "[BugFix] playlist.c DIRCACHE stop scanning when changing indices" build from 2023-01-12, which doesn't have the data abort bug, can play files from large directories and navigate their content without any speed issue.