Rockbox Technical Forums
Support and General Use => Recording => Topic started by: petur on October 03, 2006, 05:05:47 AM
-
a recent recording made with current CVS has (up till now, I'm listening at it now) at least two times missing samples (quite a number of them even)
So be warned that this can happen at the moment!
edit: Code has been changed, testers wanted :) - all seems well
-
Oi!
Let me know if I can be of assistance with regard to this matter.
:(
-
you can always test and confirm, because I've done one or two good recordings with that version (I think - I'm changing versions too much)
glitches appear at a multiple of the buffer fill time, look here:
http://www.rockbox.org/tracker/task/6109
-
Can someone report back when stabilised?
-
sure...
not yet!
-
I had the same problem recording from optical in to 320kbps mp3 with AGC on.
I've turned AGC off and gone down to 196kbps mp3 and the problem has gone away.
You could always try one of the above to see if that solves you're problem in the meanwhile.
Mike
-
Is there is some way we can standardize a recording stability test in current CVS versions for this type of problem? Maybe start a thread reporting the stability of current versions so people needing reliable versions don't end up missing important recordings?
Petur, how long into the recording did you notice dropped samples?
-
I made one of 30 minutes and one of 90 minutes with no problems at all, and then one of 90 minutes where there were missing samples after 5 minutes already...
I'm beginning to think it's a glitch: my disk is getting full (6GB left) and is probably highly fragmented (installing several new rockbox versions almost every day). So it may have needed just a bit more time to save the buffer than the buffering system clould handle.
I'm looking forward to testing Mike's patch when he gets everything working
-
This leads to the question whether the recording buffer flush watermark should be configurable in a similar way as the anti-skip buffer for playback.
-
For one thing you shouldn't skip around with your player while recording! Â ;D
The stereo WavPack at 88.2 flushes in about 1sec when it reaches the panic point (1s remaining). This is the only format and frequency which I have seen reach this point but if somehow flushing is falling behind this will prevent an overflow. There's plenty of slack space in the PCM buffer (11s at 44.1 kHz) for audio data to build up. This could adapt to pcm frequency and allow more slack at high rates but doesn't appear to be needed. I've recorded 45 minute 88.2kHz stereo wavpacks without any loss. They can't play back but that's a playback problem. The files decode correctly with FooBar2000.
If the player is shaking vigorously that can prevent ATA from writing data but I was walking around recording radio when I recorded those files so some modest movement should be allright.
EDIT: I'll mention as well that the drive spinup time is taken into account now. The remaining time to flush is 5sec + spinup time. Before a spinup might take 3s leaving only 2s remaining before overflow.
-
Is there any news regarding the recording stability of the latest daily builds?
thanks
jp
-
Yes, code changed on Nov 5. There's a couple issues:
1) Will be fixed in a day or so: After recording with S/PDIF on, then turning it off in the Playback settings, it will keep turning back on again when playing files.
2) Not sure what sample rate should be displayed for S/PDIF in the status bar: the rate recording measured when first started, the rate the encoder is using, or the current live rate. Under certain conditions all three can be quite different. I changed it to display the rate last measured and recording is assuming it still is the rate it measured when initializing. You could enter the screen at 96kHz with mp3 as the format, mp3 will have to use 48kHz since that's the highest sample rate it supports. The input could then go to say, 44.1Khz. You'll have three different rates being used in different parts. The digital rates will be rounded to the nearest the encoder accepts with a bias toward a higher one if it's right in between and nothing can be done about that. WAV and WavPack will accept any rate at all though.
I plan on adding the ability of codecs to switch and hopefully even do file splits (not sample-accurate, that's impossible) when the spdif rate changes in mid stream.
-
Not sure what sample rate should be displayed for S/PDIF in the status bar: the rate recording measured when first started, the rate the encoder is using, or the current live rate.
Prior to the filesplit feature I vote for simply showing the original incoming samplerate in the recording statusbar. If it changes during recording maybe it should blink? Thus promting the user to stop (at that point the statusbar would display the correct samplerate) and restart the recording.
-------------------------------------------------------------------------------------------------
WRS idea 1:
Adding another line to the WRS (while recording screen) displaying the currently selected input source would solve this confusion. During optical input it could also show the incoming samplerate (live).
Leaving the Samplerate in the statusbar to show the encoder's samplerate (that's what it shows when not recording from optical, right?) and showing the live samplerate in the new WRS line.
This way the recording statusbar behavior stays consistent across all inputs.
This idea presumes the filesplit-on-samplerate-change feature is implemented. Once it is I would no longer see the benefit of just showing the original samplerate.
-------------------------------------------------------------------------------------------------
WRS idea 2:
I have an idea to make all of this consistent across all targets *and* remotes!
"Hide" the Size and Filename lines behind the (new) Input and Time lines.
When the Volume line is selected (the top-most selectable line) and the user presses and holds the "UP" button, the Input and Time lines would be swapped on the display! On release of the "UP" button the display would return to normal.
This would *increase* the information displayed in the WRS by one line yet *reduce* the number of lines displayed in the WRS at any one time thus making it fit better on smaller displays.
I personally. Love this idea as it hide what I consider to be "less than essential" information from me during recording, (Filesize and Filename) yet doesn't remove it should I need to see it.
-
well I used koshs version from early october with the SWCODEC patch.
from what I get the issues were not present at low bitrates.
I recorded hours and hours in mp3 at 128 and 192 stereo.
works like a charm. I especially dig AGC.
I'll try to see what I can do to test the update from CVS.
As I only have a H300, the source display isnt an issue.
Nor is the remote, as normally during band rehearsals I just start the recording with AGC set to live (slow), and then after the rehearsal I just shut it off.
Also, I dont use the remote to record. And I dont think I will (ever). ;)
-
This weekend I did another long recording (yes it was a long time ago), and I've spotted two glitches in 70 minutes so far. Definitely better as last time but still unacceptable.
Be warned.
-
I have several loooong recordings of FM Radio I did for a stability test of the new recording code.
I am not willing to listen to hours upon hours of (boring) FM radio.
I have scanned the waveform visually using Audacity and find no drop outs.
Question: Is there a program that will find contiguous zero samples? Like if 10 or more samples are zero then inform the user somehow. I'd like to scan my test files for dropped samples, but want to do it in a semi-automated fashion.
Please let me know...
-
The missing samples are not shown as zero samples. They are just missing. As if you would have cut out a part of them (like in Audacity)
My 1:44 hour recording has 3 glitches, two of them are even more weird as last time: a part of the recording is repeated before it moves on to where it should have been.
-
So the only way for me to test this is to listen? ???
-
AHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
I'm a bit frustrated and fear the problem lies in some obscure bug not nescessarilty directly in the recording code that will take some mega effort to track down. There was a symptom on the H120 where having the callback pointer in IRAM would cause a freeze within about 15s. Something else could be overwriting adacent memory locations. I'm not dicounting the recording code itself but I just can't find a problem and I usually find them readily. Be mindful that the recording update revealed a problem with handling memory throughout the entire system...not with recording itself. So where to move next and how to proceded I'm not really certain but really want you guys to have it working perfectly.
It not a repeat since the first has a snare fill and the second doesn't and the instruments are playing differently for each section.
-
Ok, right now I'm just gonna let radio record for a couple hours and see.
-
Just got done listening to 2h 14m 25s of flawless wav recording with a voice file present and AGC safety clip. Voice restarted in the menus as expected and no crash.
Thought I had a glitch at 1:09:23 but it turned out it was just the words f***in nothin' censored in a odd way (Metallica). F***in nothin' about summs it up though.
Petur: I can send you the 1.32GB file to prove it if you like. :) Have you considered isolating your DAP from vibration? I don't doubt the recording environment must be kinda loud. Just a thought.
What I hear in your recording is exactly the sort of thing that should happen if it writing is blocked and the output fills: it _must_ drop data since it has nowhere to put it.
I suggest a lengthy recording in a quiet envoronment like just recording radio with your usual setup. If that is ok, try recording with the DAP and speakers in a box and cranking up the volume to a concert level with the corresponding low end. See what happens.
These long recordings mean a lot of trips around all the buffers and you got five glitches and I got none. I would expect something to show up for me if the recording itself were faulty but I'll keep an open mind. This problem could be unique to the H300 series but that would rule out recording itself since aside from SPDIF the code is identical to yours.
The crashes at stop are certainly another issue (since all data is written including the headers which are the very last thing updated before closing a file) as is the AGC crash (DivX0) reported elsewhere.
Sorry for being repetitve but I want my analysis well documented. ;D
-
How about recording a sine wave or something equally constant?
This way a glitch will probably be clearly visible.
How long is the drop-out? A few samples or a few seconds?
A serious test would be to record with the DAP lying still for a period of time, and then be moving in some disturbing way for an equal amount of time, and so on back and forth with the different circumstances well documented for comparison with the recorded result...
-
The skips petur had are exactly on chunk (2048 sample) boundaries and clearly discernable, both by ear and in my audio editor at the sample level. Any skips in the radio recording would be too but nothing. The DAP was still for the entire time when I tested. Inability to flush data will drop the latest data and keep the codec write position from overrunning the read (where disk writing start) position. Not preventing that would just crash it and destroy the markers for file starts and ends.
EDIT: I should also mention that all skips were forward in time, not random or backwards which is the correct behavior for the inability to write to disk for some reason recording would have no control over.
-
except for the last two where 2 seconds recording was saved twice...
fyi, I've done many recordings (over a year time) in the same conditions or even worse (too load, too much bass so more vibrations). Never had a glitch. The problems started after the new recording framework was committed though the first recordings were fine.
My bet is more on disk fragmentation. I've defragged now and will try to keep it that way.
-
Inability to flush data will drop the latest data and keep the codec write position from overrunning the read (where disk writing start) position.
Just a little idea: Could not the buffer dropping event be logged using logf to be able to narrow down possible reasons of the glitches?
-
just a quick question, been following the problems you're having with recording recently. What is the last dated CVS build that you would recommend installing if (like me) you are wanting perfect recordings without these reported glitches.
thanks.
-
Just a little idea: Could not the buffer dropping event be logged using logf to be able  to narrow down possible reasons of the glitches?
It already does.
EDIT: And I've never seen the message logged even running WavPack at 88.2 which requires a thread priority boost (in addition to CPU boost) to flush. The prio boost causes almore instant completion of data writing.
-
just a quick question, been following the problems you're having with recording recently. What is the last dated CVS build that you would recommend installing if (like me) you are wanting perfect recordings without these reported glitches.
thanks.
I don't think you'll have any problem with the latest. As I said, I did a concert length recording on my H120 and had not a single skip.
The only possible output buffer full glitch would be from the file apis being unable to write. There is no issue with any encoders being able to keep up with the data on the input end. WAV and AIFF never boost at any samplerate, even 88.2 kHz.
-
just found a way to make it freeze:
from the rec screen:
- pres rec to start recording
- press stop
- press rec again while still flushing
*bingo*
-
I can't get it to do it (freeze permanently that is). It just sticks for a second or two then just stops recording as it should. These are just the sort of operations I performed when preparing the update but will look further. The recording screen probably isn't quite set up for asynchronous stops. The H120 has no RTC and create_datetime_filename might have something to do with it since it waits until the next second before returning the file name. My x5 is in the shop so I can't test this with the RTC :-[.
While stopping, the state is still "recording" so the record button should be requesting a file split. The split should be rejected in audio_new_file because the stop should be waited for first and you can't do audio_new_file if not already recording.
-
I'm thinking about designing another type of queue that lets the waiting thread know that it's particular instance of its message in the queue is processed. The audio_* recording functions are not safe in this regard since a sequence like stop->other messages->stop would result in the thread waiting on the last stop being released when the first stop is processed.
-
Stil brainstorming... ;D
Having a synchronous "int queue_send(int msg, void *ev_data)" would eliminate the extra private queue in playback.c as well that is responsible for the resume deadlocks. The audio thread can just do queue_reply(value) which would be a no-op if the last dequeued message was posted.
Well, it's more complicated than that but this mechanism appears to be nescessary to release threads at the proper time/order they waited. Yes, KISS, but not _too_ simple.
-
Do check out and test http://www.rockbox.org/tracker/?getfile=13213
Flyspray URL:
http://www.rockbox.org/tracker/task/6109
I'm the most confident I've been so far in a change to make recording more durable. Hopefully it will close this issue.
-
Sounds great. To be tested tomorrow at a concert ;)
-
I'm not feeling the pressure! ;D
I appreciate your putting your valuable material on the line to check things out.
-
Fact is that I have little confidence in either version so I might as well try the latest fix. If it turns out to work reliable it will give me confidence and less worries while taping ;) And we do indeed need to get this thing done...
-
Guys I've been way to busy to help out lately and for that I really do apologize.
But this sounds like it must be tested and I am willing to do some tests.
Unfortunately my VMWare dev environment was recently overwritten and I haven't had time to set it back up again.
Would it be too much to ask for a custom H120 build with this patch included?
And is the only way to test for dropped samples to listening back for hours? (I'm willing to do it but I'd like to know if there is a faster way)
Again, sorry for my absence. :-\
-
Hey Davide!
Yeah, I could do that. Is an email attachment ok? I'll get it out to you ASAP.
As far as dropped samples, no, not any more. If any overflows occur you'll get a warning flashing over the recorded time display which should tell you. Though, it could come up falsly depending on timing but would be quite a narrow escape at that. If you see it, take the number down and report back. It will persist until you leave the screen or start a new recording after stopping.
-
Fact is that I have little confidence in either version so I might as well try the latest fix. If it turns out to work reliable it will give me confidence and less worries while taping ;) And we do indeed need to get this thing done...
That was intended as irony. Actually I'm biting nails and drinking too much coffee waiting for hopefully good news about this. :o
-
Current code seems to be rock stable. Thanks jhMikeS for fixing!
Case closed
-
Your welcome. I'm just sorry I didn't fix it sooner but it's just another instance of identifying the problem and a solution being obvious by stepping away from it and for awhile. I just hope it will stay fixed all by itself. Important features and refinements will be the order of business from now on I pray. 8-)
-
Aye, i've been testing and can't break recording. Thanks Mike!
-
Hello everyone,
I have most probably encountered a skip (dropped samples) on a recording I made last Saturday. Have to compare with another source to be sure, but in my ears it sounds as if....
I have a H140 and the build is of 06.12.2006 (December 6th). Approximately half of the disc space is in use.
I ran a defrag from windows recently.
I have tried to read this thread, but in between things gets very detailed and technical ???, so I only ask, if this issue is fixed now?
Are there any other advices.....?? I have HATED skips since the early days of cdr-burning.
Thanks a million
Per in Denmark
EDIT: I recorded via optical in, as I had my SBM-1 with me.
EDIT2: The build is 061207, ~ Dec 7th, 2006 (had to find my reading glasses.....)
-
For one, sorry for getting real technical about the internal details ;D For another, a threading issue was addressed in February as you can see in the previous messages. Please update. Stability was verified for that in the field and in buffer checks.
-
I have a H140 and the build is of 06.12.2006 (December 6th). Approximately half of the disc space is in use.
EDIT2: The build is 061207, ~ Dec 7th, 2006 (had to find my reading glasses.....)
I can only join jhMikeS: update....
-
Mike, I get a bit worried when I read
http://taperssection.com/index.php/topic,83004.msg1117129.html#msg1117129 and
http://taperssection.com/index.php/topic,84121.msg1117466.html#msg1117466
It seems like the person who recently had a warning displayed on his display was using a svn version of 7 March 2007, not the REP..... (confirmation pending)
So far all _my_ recordings have been fine.
-
Petur: Was going to mention before you ran out of IRC that I do know of trouble spots in the scheduler itself...I suppose I shouldn't delay repairs there and I'll give each bit of the threading a thorough exam. It's likely the cause of the blocking violations and perhaps this one as well.
-
I can only join jhMikeS: update....
Done yesterday!
Stability was verified for that in the field and in buffer checks.
What can I do to test? Just record and listen closely afterwards? Are there any messages on the screen if troubles occur?
How can I check buffer?
Thanks for all your struggle 8-)
Per in Denmark
-
Warning: 0x00000001 -> PCM buffer overflow
Warning: 0x00000002 -> Encoder buffer overflow
Warning: 0x00000003 -> Both of those
Flashed over the time display.
-
petur: also, I myself am confusing the two warnings...when I'm just not awake yet. :o Well, no matter, Semayat had an encoder buffer overflow which could still be caused by that. Both iRivers have backlight fading? This has caused the scheduler to get borked on my H120 recording and must be due to the known scheduler issue. Well, I'm on that one since a solid base for everything must be in place since corrupting the scheduler with IRQ posts can kill threads permanently. That would also explain the inability to shut down since pcmrec must work for shutdown to proceed.
-
Petur: Was going to mention before you ran out of IRC that I do know of trouble spots in the scheduler itself...I suppose I shouldn't delay repairs there and I'll give each bit of the threading a thorough exam. It's likely the cause of the blocking violations and perhaps this one as well.
Ah... ok... and sorry for running away just like that, things are getting a bit hectic at work...
Both iRivers have backlight fading? This has caused the scheduler to get borked on my H120 recording and must be due to the known scheduler issue.
of the irivers, only the h1x0 has fading, the h3x0 uses a different type of backlight control that has less steps.
-
Warning: 0x00000001 -> PCM buffer overflow
Warning: 0x00000002 -> Encoder buffer overflow
Warning: 0x00000003 -> Both of those
Flashed over the time display.
For some seconds or will warning(s) be visible when the show is over and I take the iRiver out of my pocket? I usually do stealth recordings and when gain is set I park the iRiver in my pocket.
I only check it again if soundlevel at venue increases dangerously :D
Per in Denmark
-
The warnings stick until:
1) You leave the recording screen :)
2) You press stop and then start a new recording.
-
Petur, here's an old flyspray task you might be interested in that strongly suggests the corruption on full disk is not a part of the recording code at all and is not new.
Reported 9 days before the introduction of codec based recording:
265 days old: Running out of space corrupts recorded file
http://www.rockbox.org/tracker/5852
-
Petur, here's an old flyspray task you might be interested in that strongly suggests the corruption on full disk is not a part of the recording code at all and is not new.
Reported 9 days before the introduction of codec based recording:
265 days old: Running out of space corrupts recorded file
http://www.rockbox.org/tracker/5852
Thanks... I'll try to find some time to dive into that code (but most of coming week is already booked by real life)