How are you moving your Windows 10 Lace computers to new versions of Windows 10 in a GxP lab?
Lab computer systems seem to get little attention from IT and even less love. Our validated Lace machines are complex to setup and critical to helping departments get work done. We are facing a new hurdle with Windows 10 with end of support (EOS) where Microsoft will no longer release security updates.
There was a major push to get all Windows 7 computers off the network this year. January 2020 was the EOS and final push to remove legacy W7 from the network.
Rant:
Windows 10 is different. W10 has two yearly *versions*. Each is released with only 12 - 30 months of support. Security patches are released every month during this life-cycle. When a version of W10 goes out of support (EOS) there are no further security patches or support from MS.
Background:
Since the 2015 release of Windows 10, Microsoft has significantly changed its operating system (OS) strategy for product updates. Microsoft propensity now is to push significant Windows 10 updates via Windows Update, and to eventually force updates. Windows-as-a-Service approach (WaaS).. The monthly critical security patches are an expected and excepted balance. When they can force a new W10 OS we have to ensure our validated applications function with the labs timelines.
Rant:
IT is not going to allow old EOS W10 to remain on the network! How are we going to address this every 1 - 2 years? I do not want to have a flood of paperwork or downtime! We deployed W10 (1709) which is EOS in October 2020. We have to get dozens of machines upgraded before November. If I have to do all the same IQ/OQ/PQ procedures for each one there will be a large risk of downtime. Of course you should test first and we should have Waters specify their support.
Proposal:
My suggestion is to treat the new W10 release in a similar way to the monthly patches. Compatibility is what MS is being paid for so Empower should continue to work as before. The IQ/OQ should not be impacted. I don't believe the FDA wants older unpatched nodes risking cybersecurity breaches. So all vendor updates could be considered like-for-like and less risk overall. Higher risk if they are NOT done.
Questions:
Microsoft's Windows 10 release support policy impacts very short End-of-life and End-of-support details. We are required to install critical security patches each month.
What GxP validated paperwork does that generate on your site?
If you have Windows 10 on your Lace nodes how are you planning to keep W10 updated?
What do you think?
0
Answers
-
So... I would try to take a different approach. The risk is the EOL product will no longer be maintained / patched by MS. Clearly a security IT risk.
So, why not setup a VLAN? No outside world communication EXCEPT to the Empower / Citrix servers. Communications can be configured by both direction and port on a VLAN. So, why not isolate your internal instrument network?
I am curious to hear thoughts on the VLAN approach. This is going to need to be a war I wage with my IT department as well.0 -
Maybe for a small site not updating security patches could be an option. For mid size and larger there really is no path except to stay current. The connected world is only getting bigger; the Empower app will also continue to be patched and gain features. The FDA is going to see our plan to guard against cyber hacks vs the risk of installing vendor supplied security patches. The "least burdensome way" has to consider the whole system environment.
I helped deploy an isolated lab vlan that had EOL custom machines. IT would not allow these on the 'business' network. Used a $$ firewall to allow limited port access. We still needed some access! It does work however it was a lot of work, not an easy way. Most will need remote support and data backup etc.
I really hope more folks will add any thoughts for deeper discussion. Instead of a solution (isolated vlan could work) what are your sites doing now? If LTSB was deployed, how is it holding up? When it has to be replaced will you have to wipe and reinstall? I suspect you can pay the piper now, or later by staying current with new W10 releases.
What is the best practice?0 -
That's the thing, I work for one of the larger Pharma companies and have gotten an incredible amount of push back from the VLAN approach from IT. I get it's time and money, but I am not seeing any other way to both efficiently and economically manage.
To my knowledge we have not deployed LTSB and are not doing so for my own site's FR5 upgrade rolling out later this month.
I am curious to hear the opinions of others as this just seems like a lose-lose no matter what path you choose.0 -
I hope we can find win-win! Lab Lace systems are not like end user workstations with Office apps. It may be worth the fight to set guidelines with IT and the network teams so we can keep getting the work done!
I sometimes shake my head at some of the (willful?) lack of IT support in the lab. "We don't have time for validated systems; take it or leave it; I don't do that." What do we need? For this thread I suggest: staying patched = stay connected and secure.
Windows 7 had a very long (+10 year) run where we just let the old hardware and monthly patches be normal. Old, slow, breaks more often, harder to replace..
Microsoft is forcing a very short life cycle with W10. I think that IT will come down with a deadline to get W10 EOS machines off the network.- How do we stay validated (NOW) with monthly MS security patches?
- How do we stay validated with Windows 10 releases? (Work SMARTER)
- I have dozens of W10 Lace that will be EOS next month!
0 -
To be honest, this actually borders on a deep rooted, you shouldn't be doing that concern I've had for years and years.
Relevant to this topic are items such as GAMP and the fact that Waters lists the Windows patches that they have tested Empower against. Note that the disclaimer from Waters is "we have tested extensively with these windows updates. That does not mean other patches won't work, but it does mean we have not tested them".
Keeping the previous paragraph in mind, I am of the opinion that if you patch the lace beyond what Empower was originally qualified on or beyond what Waters has personally tested and evaluated, then you are in no man's land (re: an unvalidated and untested state).
Waters developed the SQT process to be fast and efficient. SQT is backed by internal testing at corporate along the lines of: can I run two of these instruments at once; does vial 1 really mean vial one; can I collect data from 1-80 Hz. Every piece of functionality is tested.
A patch might break that. Say a windows update gets pushed out that changes something (maybe CMOS or network functionality). Say you have a lac/e connected to an e-satin that previously could buffer for 72+ hours. Now, because of the patch the LAC/E typically fails the buffering test in 13-24 hours.
Say maybe another problem comes up where the lac/e is able to buffer with no network connection, but windows checks the time against a time server. Say a patch changed this so windows will check the time against the CMOS if a time sever is unavailable. Maybe this happens while the lac/e is buffering and the CMOS is off by a few min or seconds. Whola! Oracle now rejects the buffered data.
Both of these two instances are very real and have happened to us first hand.
This is why you patch to the extent Waters lists, and isolate on a VLAN.
Sadly, the company that I work for is so big it reacts to everything and probably doesn't have visibility to what is going on with Win10 and support. I would imagine being in the same boat we are in with Win7 now for Win10 in 1-2 years.
I'd like to see this not happen, which is why I am screaming at the forrest. However, the trees are not listening.
0 -
The bottom line of this is the question:
How do you roll out patches / Updates and how do you keep your LAC/e boxes in a validated status without ending in a deadly amount of testing and documentation efforts.
We do patch by using a dedicated environment for this where we are having full control of what is being deployed.
In addition, we do install LAC/e boxes by deploying pre-validated Images to the boxes.
This allows us (if needed) to update our more than 900 LAC/e boxes in less than 2 hours in a big bang.
So any newly installed LAC/e is a one-one copy of any other LAC/e in the network.
We do centrally test new images by deploying it to test boxes where we then run defined tests to ensure instruments are working as expected.
After the successful release of the image for use, we then deploy.
THat way we are able to reduce the effort for testing to approx. 1 or 2 weeks.1
