What to do with "Project integrity" ?

Syntaxerror

<p>Hi evrybody. Here is my first question on this forum.</p><p></p><p>After backing up project, a "Project Integrity Test" is done. OK when no error are found but sometime the integrity reports some error about some raw data file. The following message appears : "Error: Failed to read good raw datafile with ID XXXX"</p><p>So my question is short : are they lost ? If not, how can I retrieve ?</p><p></p><p>Thanx to the community</p>

kjbu

Hi!

I get this error on 50% of my restored projects (when upgrade from 1154 project version).

I have the same question as you.

- kjbu

Dot

Hello

The Project Integrity errors you see indicate there may be a mismatch between the file and the database.

Raw Data Integrity ensures that for each Channel ID in a given project a Raw Data File Integrity check is performed. Raw Data File Integrity ensure that for each Channel ID given, a raw data file exists and can be read and it’s checksum is OK.

You might want open the project then go to the channels. There right click in the white space and select Show All Columns. If there is a row in the channel table that is not found in the project's raw data directory, then perhaps data is still on the acquisition node where it was collected? You might check the recover.log file on the acquisition node.

If the data is stuck on the acquisition node, then most likely the channel id was not populated. In some cases, the first channel during buffering will be in the table but the file will not be on the file server. So showing the Channel ID column, using the list of channels that failed verification and checking to see if they are present might be helpful here. If they are, then open those in review and see if you can do review incomplete data.

If they are neither in the raw data folder nor on the acquisition node, then you should look at the backups of the server. If they are nowhere, then there isn't much anything else to do.

If you need assistance you might wish to contact your local service organization.

I hope this is helpful.

Dot

reinhold

Hi,

we also see these messages and I agree with Dot.

We have the feeling that in most cases we have these "missing" files when we start a single injection and abort it.

I think, Empower creates an entry in the channel table but does not create the RAW file.

We normally ignore the message.

Reinhold

Ratoshigan

We see them all the time as well.  Nearly all we have seen so far have been because of what Reinhold mentioned.  Once Empower sets up for an injection, the channel IDs have been created.  If the injection never happens (aborted run, instrument failure or non-controlled autosampler not programmed to inject anything), there is no file for that channel ID.  Empower doesn't delete the channel ID once one has been assigned, so it reports it as a project integrity error (Error: Failed to read good raw datafile with ID XXXX).

After our first archive at at my company, we documented the occurrance and accepted it as a non-issue, but we still verify that the error was due to this problem.  To verify, I usually go to the channels tab (display channel ID to find the XXXX number that corresponds).  Right click on the channel that is missing and view as sample sets (if there isn't a sample set, it was a single injection more than likely stopped before injection).  Then right click on the sample set and view as channels.  The missing channel will nearly always be the last injection of the sample set.  If you try to review that channel, it should give you a "data missing" error in review.  They can show up in the middle of a sample set if someone sees a run error, fixes it and continues the run.  You will usually be able to see a timestamp gap between injections or at the very least a duplicate injection name.

kjbu

I think that Waters should remove this function.

Why report errors when in fact there are no errors?

Or fix the function not to include these common 'non-errors'.

Anybody agree on that?

Ratoshigan

I don't think it should be removed.  It was useful for detecting modifications to files we had occur during an incident archiving.

I would rather see it not pause the archive so that you can acknowledge every project integrity failure.  There's nothing worse than having to babysit a several hour process.  Either generate a popup list of all the projects that had project integrity errors at the end of the archive or not at all since it already logs them in the message center.

yat74

I agree, having to acknowledge the "project integrity" error messege does make the backup process tedious.  A summary table at the end of the archival process process would be nice. 

GPK

I agree as well. The project backup process is incredibly tedious when having to acknowledge all of these "errors".  Provide a summary at the end but don't suspend the process each time it encounters an error.

peteck2007

I do agree also. Restoring lots of projects does take a lot of time - in our network, restoring a whole year of data would take up to 36 hours. Nothing is more anoying, when you come to the computer at 8am just to see that it is stuck on an integrity error since 5pm the day before, just 5 minutes after you left the office...

THNDRacket

Indeed. Nothing more frustrating than starting a backup and coming in the next day to see it stopped on the first project awaiting me to acknowledge an integrity error on a data file for an injection that was begun but aborted. I keep hoping Waters comes up with a hotfix for that.

GPK

I agree as well.  The integrity check is somewhat useful but when backing up hundreds of projects the current design isn't good.  Changes to this design are coming in Empower 3 from my understanding.   This needs to be changed in Empower 2.

GPK

Perhaps a better idea - develop a process to check these projects routinely and force the users to acknowledge the error rather than an unlucky administrator so that the errors aren't detected one year later when we archive them.  No one wants to find out they had a problem with a datafile a year later.

kenny_78

Not something I've had to do often, but I agree it'd be useful to have the option of summarising the errors after the process rather than having to respond to each one as it happens. I've asked their Toolkit expert if a workaround could be produced (http://forums.waters.com/community/informatics/blog/2010/02/09/the-empower-toolkit-what-is-it-for-and-why-you-might-be-interested#comments) but it looks as though the option would be more likely to appear in a feature release or future version of Empower.

AValerius

Back to the original question!

There could be different reasons:

1. Data mismatch in review window instead of a chromatogram
   • Seems to be a corrupt (may be not complete) raw file. You can compare the backed raw file with the imported one. I don't know if you could do anything else to solve this.
     
2. Data missing in review window instead of a chromatogram
   • The raw data file ist not available. In most cases this is due to a user abort of acquisition or user abort of backup. There is no raw data file available. Very seldom we had the case the backup function didn't backup all raw data files. Don't know why. This is sporadic and not reproducable till now.
     
3. Data incomplete in review window instead of a chromatogram
   • This might be the case due to importing older projects which could cause troubles with the internal checksum function. In this case you can do a verify of the related chromatogram. I can not remember exactly where you will find this verify option. May be you wil find an additional entry "Verify..." in the context menu of the chromatogram chart. Just try a right mouse click and see what you will get. In our environment it will only be shown to administrators.
     

Regarding the integrity check within the backup function:

I like this integrity check, but Waters should make the integrity check silent. without a popup.

Anyway we use Toolkit functionality to backup projects, check the last row of the integrity log and additionally check the number and size of raw files afterwards independently to the integrity function.

Arjen

jeanette

Hi

We have also seen this kind of message when migrating projects from older versions into Empower 2. I think it is a very nice feature that Empower perform this check.

Normally we have only seen this message if in fact data were missing "Data missing" if you open in review.

Last time when we restored Empower 1 projects into Empower 2 we had 8-10 of tgese message and all of them was on files with results stored - which told us that the data file must have been OK at some point. Talking to Waters DK I was told to try to verify incomplete data manually.

This is taken from Help:

If the data file is incomplete, Empower displays "Data incomplete" on the chromatogram plot. If you have the Verify incomplete data in Raw Data Files privilege, you can right-click on the chromatogram, and click Verify incomplete data. Empower attempts to display the Data file with as much information as possible. If Empower can display the Data file, you can save it and use it as normal. If the Data file is unacceptable, click Cancel, so the data file will remain incomplete,  and no changes will be made.