Recently Hewlett-Packard invited a group of bloggers into their Houston testing labs. Why do that? That was my question going on the tour – but it was quickly apparent that HP has a lot to be proud of in an area that adds tremendous value to their computers. Let’s take a look with HP at the Science of PC!
Before I get into the actual tour, I have to comment that I found it amazing from start to finish that HP was able to assemble such a tour! I mean, these were active labs we toured with test technicians and engineers and current-generation products and test equipment … and none of us had full NDAs that we were required to sign!
Having been in the equipment and semiconductor industries in the past and currently working for a company where intellectual property is very important – I was really struck by the efforts HP made at accessibility and disclosure. Of course, some visitors wanted to see more cutting edge stuff and were disappointed … but I knew better.
Kicking off the tour and setting the scene – we would have access to the actual people doing the work and see what they do.
Quick View – What I learned
If I had to list my five biggest learnings, here is what struck me the most:
- Integration – product testing and quality are ‘in HP’s DNA’, as the saying goes. It is what allowed Compaq to quickly become the top PC maker, and for HP to build such a legacy of great products. It means that quality is considered throughout the design cycle – something more difficult when testing is outsourced like many PC makers are doing now.
- State-of-the-Art Focus – this is different from ‘bleeding edge’, with more of a focus on understanding what is going to add value to their core customers and how to ensure it will work properly in any number of environments.
- Technical Breadth – we met with a number of really smart people of all ages and from all types of backgrounds, people who know the science and engineering but also possess the context to apply what they have learned.
- Passion for Quality – this sounds like a buzzword but it is much more than that. The science of quality and reliability is critical to making products that will withstand the test of time, and having people who have quality and reliability as part of their mantra means that new products will reflect learnings from old failures.
- Global Reach – with testing a truly global endeavour for HP, the level of integration and cross-communication evident in their operations is amazing. The work of building this into their workstreams has paid off as markets have shifted over the last few decades.
Just some of the stuff being powered by the ZBook!
HP Houston Labs – The Science of PC
The HP Houston Lab facility is part of the old Compaq complex, and Compaq’s history of quality is what allowed them to become incredibly successful as the first maker of ‘IBM clones’ back in the 80s.
One of the first people we met on the tour was Steve Prato, who was working on Thunderbolt 2 testing and certification. He showed us a Thunderbolt 2 daisy chain set with 8 drives on the cable. Thunderbolt 2 is high performance and low latency, offering the ability for simultaneous multiple-streaming.
Multiple daisy-chained monitors
He showed the Thunderbolt 2 streaming working on the HP ZBook 15 and 17 – and noted that it wasn’t supported on the ‘Ultrabook’ form-factor ZBook 14 due to the space constraints of getting the daughtercard on the motherboard.
In order to get Thunderbolt into their systems, they need to license the name from Intel and get their products independently certified. This allows them to start with a small offering and evolve into other products over time as they gain more experience. They have already noted interoperability issues between Thunderbolt 1 and 2, which can cause problems and is a reminder that not all standards are standard.
Steve Hlawek took us further into the testing of the ZBook 15 and 17, looking at the 4k display (gorgeous!) and the Mosaic Multi-display. He showed us a stress test with 2 Intel and 3 Nvidia graphics processors powering 5 displays simultaneously streaming video over Thunderbolt; impressive stuff.
Then we were led through software validation by Pam who heads the international team (lost her last name, sorry!). One of their big metrics is that there is an estimated 133,000 hours of test before release.
Showing off how the new drive-based encryption works
They start with component level test – some of this is done at suppliers with verification by HP. There was discussion about validation, with Pam talking about sampling and having a long-term history of what types of components are susceptible to which kinds of failures – and what that means for failure analysis and testing. As systems have moved away from mechanical movement (hard drives and optical disks) to more densely packed circuits, the failures have evolved and so has HP’s testing and analysis.
This is a ‘wall of patents’ – loads of innovation in Houston!
Once the systems begin to get integrated, things get interesting – the team is divided between Houston and locations in Asia, with each having different responsibilities but all working towards the same goals. ‘WAT’ is the term for their integration test efforts, and they are handled through solutions tests with the ISV to make sure subsystems work.
The products that have come through here – note the Compaq heritage, no early HP stuff here.
Drivers are where things start to get tricky (as any Windows user can attest!) – HP has their components and subsystems tested individually with drivers, then as they are fully integrated into subsystems, and finally as complete product builds. What they have seen is that most of the components and subsystems have matured to the point where they are actually quite reliable.
Inside an acoustic test lab
Once they are well on the way with their product development they need to get the WHQL certification from Microsoft, which is a mandatory requirement for Windows certification.
The whole process can take about 9 months, and involves a large cross-functional team with leads for each component. There are 10-12 total leads, and then there are integration teams which are all in Asia. It is an incredibly complex cross-function effort that eventually leads to products on shelves.
Inside the acoustic test lab
Data protection has become more critical with each passing year – both in terms of how much critical data we handle and how much of it has potential exposure on the internet. Gordon Cross talked about how prior methods were software based and inherently slowed down the system (personally I remember having data transfers to flash drives taking minutes-per-megabyte), so HP has been working on other methods.
Hard drive shock and vibration tests
The current scheme is a multi-layer hybrid with the control based in the drive itself. It has an encryption key exposed to the BIOS so it is secure and robust to system failure.
Crush test for hard drives
Khalid Hameed told us about the HP Stratus, which features the ability to utilize the HP Sure Start technology to detect and correct BIOS corruption. It can use the HP server or a custom corporate server to bring the BIOS back up to date and correct any hardware configuration issues related to the corruption. This all happens independent of the operating system and has no visibility to the actual data on the system, meaning it cannot be used to expose sensitive data.
Box crush testing
Before we went on an extended tour of the environmental test facilities, Jimmy Holbert gave us some vision of how all of this fits together in the context of HP’s ZBook products. The ZBook is really a mobile workstation and is made in a collaborative effort between the desktop workstation and Elitebook groups, with the result being a sleek business computer with the heart of a desktop replacement and at a price much better than would previously have been possible.
Environmental chamber
The ZBook series brings in the incredible Thunderbolt 2 technology with 20GB/sec throughput, but is also partially ruggedized and goes through a series of MIL STD tests for dust, spills, vibration, drops and so on. The ZBook 14 is really a separate product, and is a compromise of space and performance, built upon an ultrabook chassis.
Showing off some of the full-system and packaged test stations
The environmental test facility in Houston has a focus on getting products early according to Bruce Van Alstyne. This is not a full reliability lab, but rather a small-scale initial failure test lab that grabs a small statistical sampling of parts and products to assess likelihood of failure under a number of conditions.
Hi frequency, low frequency, bumps and bangs – and it just keeps performing!
Some of the tests they perform include packaging tests (box crush); elevation tests (up to 118,000 ft!) which deals with air thickness, impacts of cooling and how it affects optical drives; diaphragm tests for keyboards; corrosion protection tests in a salt box (by ASTM v117 standard.
Tablet stress testing
Mohammed Chowdry took us into the acoustic test labs. The chamber was amazing to enter, as when he closed the door it felt like all sound was simply sucked out of the air! The room is semi-anechoic (floor is reflective), and is used to check products according to the ISO 7779, 9296 testing standards. The lab tests everything from HP, mostly notebooks but also desktop, workstations, and servers. Servers have more fans and therefore produce more noise.
Strain measurements in subsystems
Noise issues typically come from fans and drives, but they have historically also had problems with people using cheap-unbalanced disks in their optical drives. The room has a variety of microphones and analyzers for signal and frequency analysis, all laid out and positioned according to the standards.
FT-IR lab for materials analysis
The acoustic test lab has two purposes: #1 development support, #2 for market certification, data sheet information. It has also been certified as NIST 3rd party lab, independent – gets test data in 3-4 weeks. Everything in the lab is externally calibrated and audited on a 2-year schedule.
SEM gets to 10,000X magnification!
In the rest of the lab we looked at shock, vibration and packaging tests. Floyd Privette showed off disk drive tests for shock and voibration, as well as drop tests and Instron-based crush tests. John took us through a variety of force and frequency tests for packaged products and some newer precision drop tests for tablets and laptops.
Racks of circuit boards under extreme test conditions for a very long time
EMC Test Lab
EMC stands for Electro-Magnetic Compliance, and is the ability of a product to properly control & contain radio-frequency emissions leaving or entering the device. Rick Shaw took us through the labs, and showed us some of the compliance and reliability tests they performed.
Real time X-Ray system
One of the coolest things we saw during the entire tour was the SG-64 test environment. It was a massive EMC ring superstructure in a fully anechoic chamber. It was impossible to get the whole thing in a picture, but check this out:
The SG-64 EMC Test Chamber
Yeah, I know. But trust me, it is visually impressive (and bonus geek points to anyone who can guess the SG-64 name origins). The room is an advanced setup for doing a pattern test for radio transmitters. This allows for full 3D field emissions tests in a way that wasn’t previously possible. With more and more of our access coming wirelessly, it is important to understand the strengths and weaknesses of components.
Looking inside the circuit board with the Real-Time X-Ray system
They can also check a variety of other field-emission tests including magnetic fields, ELF, TCL and conducted emissions. There is a radiated emissions lab that tests from 30MHz to 40GHz, and also a massive semi-anechoic chamber that allows for radiated emissions testing, interference analysis, and also immunity, which is a check on what can get into the system.
Failure analysis testing
Most testing of displays, servers, calculators, and administration is in Houston, with facilities in Roseville and Fort Collins to handle some high-end products and other areas of testing.
Multi-purpose robot puts systems through the paces
One of the last things we watched was basic grounding checks – a spark-generator around the casing to try to get the system to fail. Again, simple test, but really cool watching the sparks jump.
Open, close, open, close, open, close, open …
Reliability Lab
Our final visit was to the Reliability lab, where we got to look at a number of tests looking at failures, as well as material analysis and life testing.
Weighted touchscreen tester
John Olivaris walked us through the materials labs, which are used to look at products and components either in development or after failure. They have a number of tools at their disposal including a FT-IR lab with microscopy, material identification tools and libraries, and an XRF (X-Ray Fluorescence) which is used for metal layer thickness analysis.
Dual trackpad abuse system!
The lab also has a cross-sectioning microscope for failure analysis as well as a SEM (scanning electron microscope) for resolutions beyond 10,000X magnification. SEM requires a coated sample, making it more difficult to look at metallized surfaces. The lab has a large number of thermal cycling chambers that Michael Fahrenbach showed us, to accelerate any potential temperature-based failures, and also have instrumented many of the tests to observe conduction and other properties in real-time as the tests are conducted.
Inside the dust test system
Henry Garcia showed off what was my second favorite thing – a real-time X-Ray test. With this they are able to put a full laptop display on the system, scan around, and zoom in real-time. He had a display with a broken connection embedded deep in the layers that would have been very difficult to see any other way. With this system information is gathered and fed back to engineering to determine what happened and how to deal with it in the future. Really cool stuff.
In-depth materials and components testing and analysis
Other tests they perform in the lab are HALT – highly accelerated life testing, as well as an Instron for normal force displacement testing to see how products perform under normal life forces as well as more aggressive shocks and bending forces.
Output of the EMC field test system
In the Reliability Engineering Lab, John Vijil took us through some similar tests we had seen before but done here for engineering failure analysis rather than for compliance testing. They have shaker tables for hard drives, and a room for electrostatic discharge, where it took 10000V to shut down system – which then immediately restarted (in other words, it wasn’t fried!)
Constant set of test packages run as the system is subjected to various external signals
There was another room for small-scale reliability testing of interface components. There was a multi-purpose robot that performs tablet-centric tests like presses, swipes and other gestures. There was also a laptop hinge open-close tester that worked on several laptops simultaneously (for context of why this is important, Google “Nintendo DS hinge crack” or “HP 200 hinge crack”).
They also had a screen press test for laptop touchscreens, a keyboard and mouse test that rubs and looks for print wear, a displacement test for keys, and a touchpad abrasion test that puts the pad through 250,000 cycles! The lab has a number of environmental test chambers, such as a dust chamber and a thermal shock chamber that can cycle from 40C to -40C in 1 second!
Static discharge gun in use (can just see the spark jump out of the wand coming from the tester’s hand.
These labs are all testing to standards, and work together on failure analysis to do teardowns and corrosion analysis, microscopy, and so on.
Oh, and of course they really took care of us in terms of food as well!
We were very well taken care of! Thanks HP!
Conclusions
I return to my initial question – WHY would HP want us in their labs? Simple – they are representative of both the history and the leading edge of computer testing and reliability, they have stayed true to their ideals even as the market has shifted and margins have thinned, and they have a commitment to quality and reliability that makes me happy that my company has a HP laptop on my desk at work everyday.
