Recently Hewlett-Packard invited a group of bloggers into their Houston testing labs. Why do that? That was my question going on the tour – but it was quickly apparent that HP has a lot to be proud of in an area that adds tremendous value to their computers. Let’s take a look with HP at the Science of PC!
Before I get into the actual tour, I have to comment that I found it amazing from start to finish that HP was able to assemble such a tour! I mean, these were active labs we toured with test technicians and engineers and current-generation products and test equipment … and none of us had full NDAs that we were required to sign!
Having been in the equipment and semiconductor industries in the past and currently working for a company where intellectual property is very important – I was really struck by the efforts HP made at accessibility and disclosure. Of course, some visitors wanted to see more cutting edge stuff and were disappointed … but I knew better.
Quick View – What I learned
If I had to list my five biggest learnings, here is what struck me the most:
- Integration – product testing and quality are ‘in HP’s DNA’, as the saying goes. It is what allowed Compaq to quickly become the top PC maker, and for HP to build such a legacy of great products. It means that quality is considered throughout the design cycle – something more difficult when testing is outsourced like many PC makers are doing now.
- State-of-the-Art Focus – this is different from ‘bleeding edge’, with more of a focus on understanding what is going to add value to their core customers and how to ensure it will work properly in any number of environments.
- Technical Breadth – we met with a number of really smart people of all ages and from all types of backgrounds, people who know the science and engineering but also possess the context to apply what they have learned.
- Passion for Quality – this sounds like a buzzword but it is much more than that. The science of quality and reliability is critical to making products that will withstand the test of time, and having people who have quality and reliability as part of their mantra means that new products will reflect learnings from old failures.
- Global Reach – with testing a truly global endeavour for HP, the level of integration and cross-communication evident in their operations is amazing. The work of building this into their workstreams has paid off as markets have shifted over the last few decades.
HP Houston Labs – The Science of PC
The HP Houston Lab facility is part of the old Compaq complex, and Compaq’s history of quality is what allowed them to become incredibly successful as the first maker of ‘IBM clones’ back in the 80s.
One of the first people we met on the tour was Steve Prato, who was working on Thunderbolt 2 testing and certification. He showed us a Thunderbolt 2 daisy chain set with 8 drives on the cable. Thunderbolt 2 is high performance and low latency, offering the ability for simultaneous multiple-streaming.
He showed the Thunderbolt 2 streaming working on the HP ZBook 15 and 17 – and noted that it wasn’t supported on the ‘Ultrabook’ form-factor ZBook 14 due to the space constraints of getting the daughtercard on the motherboard.
In order to get Thunderbolt into their systems, they need to license the name from Intel and get their products independently certified. This allows them to start with a small offering and evolve into other products over time as they gain more experience. They have already noted interoperability issues between Thunderbolt 1 and 2, which can cause problems and is a reminder that not all standards are standard.
Steve Hlawek took us further into the testing of the ZBook 15 and 17, looking at the 4k display (gorgeous!) and the Mosaic Multi-display. He showed us a stress test with 2 Intel and 3 Nvidia graphics processors powering 5 displays simultaneously streaming video over Thunderbolt; impressive stuff.
Then we were led through software validation by Pam who heads the international team (lost her last name, sorry!). One of their big metrics is that there is an estimated 133,000 hours of test before release.
They start with component level test – some of this is done at suppliers with verification by HP. There was discussion about validation, with Pam talking about sampling and having a long-term history of what types of components are susceptible to which kinds of failures – and what that means for failure analysis and testing. As systems have moved away from mechanical movement (hard drives and optical disks) to more densely packed circuits, the failures have evolved and so has HP’s testing and analysis.
Once the systems begin to get integrated, things get interesting – the team is divided between Houston and locations in Asia, with each having different responsibilities but all working towards the same goals. ‘WAT’ is the term for their integration test efforts, and they are handled through solutions tests with the ISV to make sure subsystems work.
Drivers are where things start to get tricky (as any Windows user can attest!) – HP has their components and subsystems tested individually with drivers, then as they are fully integrated into subsystems, and finally as complete product builds. What they have seen is that most of the components and subsystems have matured to the point where they are actually quite reliable.
Once they are well on the way with their product development they need to get the WHQL certification from Microsoft, which is a mandatory requirement for Windows certification.
The whole process can take about 9 months, and involves a large cross-functional team with leads for each component. There are 10-12 total leads, and then there are integration teams which are all in Asia. It is an incredibly complex cross-function effort that eventually leads to products on shelves.
Data protection has become more critical with each passing year – both in terms of how much critical data we handle and how much of it has potential exposure on the internet. Gordon Cross talked about how prior methods were software based and inherently slowed down the system (personally I remember having data transfers to flash drives taking minutes-per-megabyte), so HP has been working on other methods.
The current scheme is a multi-layer hybrid with the control based in the drive itself. It has an encryption key exposed to the BIOS so it is secure and robust to system failure.
Khalid Hameed told us about the HP Stratus, which features the ability to utilize the HP Sure Start technology to detect and correct BIOS corruption. It can use the HP server or a custom corporate server to bring the BIOS back up to date and correct any hardware configuration issues related to the corruption. This all happens independent of the operating system and has no visibility to the actual data on the system, meaning it cannot be used to expose sensitive data.
Before we went on an extended tour of the environmental test facilities, Jimmy Holbert gave us some vision of how all of this fits together in the context of HP’s ZBook products. The ZBook is really a mobile workstation and is made in a collaborative effort between the desktop workstation and Elitebook groups, with the result being a sleek business computer with the heart of a desktop replacement and at a price much better than would previously have been possible.
The ZBook series brings in the incredible Thunderbolt 2 technology with 20GB/sec throughput, but is also partially ruggedized and goes through a series of MIL STD tests for dust, spills, vibration, drops and so on. The ZBook 14 is really a separate product, and is a compromise of space and performance, built upon an ultrabook chassis.
The environmental test facility in Houston has a focus on getting products early according to Bruce Van Alstyne. This is not a full reliability lab, but rather a small-scale initial failure test lab that grabs a small statistical sampling of parts and products to assess likelihood of failure under a number of conditions.
Some of the tests they perform include packaging tests (box crush); elevation tests (up to 118,000 ft!) which deals with air thickness, impacts of cooling and how it affects optical drives; diaphragm tests for keyboards; corrosion protection tests in a salt box (by ASTM v117 standard.
Mohammed Chowdry took us into the acoustic test labs. The chamber was amazing to enter, as when he closed the door it felt like all sound was simply sucked out of the air! The room is semi-anechoic (floor is reflective), and is used to check products according to the ISO 7779, 9296 testing standards. The lab tests everything from HP, mostly notebooks but also desktop, workstations, and servers. Servers have more fans and therefore produce more noise.
Noise issues typically come from fans and drives, but they have historically also had problems with people using cheap-unbalanced disks in their optical drives. The room has a variety of microphones and analyzers for signal and frequency analysis, all laid out and positioned according to the standards.
The acoustic test lab has two purposes: #1 development support, #2 for market certification, data sheet information. It has also been certified as NIST 3rd party lab, independent – gets test data in 3-4 weeks. Everything in the lab is externally calibrated and audited on a 2-year schedule.
In the rest of the lab we looked at shock, vibration and packaging tests. Floyd Privette showed off disk drive tests for shock and voibration, as well as drop tests and Instron-based crush tests. John took us through a variety of force and frequency tests for packaged products and some newer precision drop tests for tablets and laptops.
EMC Test Lab
EMC stands for Electro-Magnetic Compliance, and is the ability of a product to properly control & contain radio-frequency emissions leaving or entering the device. Rick Shaw took us through the labs, and showed us some of the compliance and reliability tests they performed.
One of the coolest things we saw during the entire tour was the SG-64 test environment. It was a massive EMC ring superstructure in a fully anechoic chamber. It was impossible to get the whole thing in a picture, but check this out:
Yeah, I know. But trust me, it is visually impressive (and bonus geek points to anyone who can guess the SG-64 name origins). The room is an advanced setup for doing a pattern test for radio transmitters. This allows for full 3D field emissions tests in a way that wasn’t previously possible. With more and more of our access coming wirelessly, it is important to understand the strengths and weaknesses of components.
They can also check a variety of other field-emission tests including magnetic fields, ELF, TCL and conducted emissions. There is a radiated emissions lab that tests from 30MHz to 40GHz, and also a massive semi-anechoic chamber that allows for radiated emissions testing, interference analysis, and also immunity, which is a check on what can get into the system.
Most testing of displays, servers, calculators, and administration is in Houston, with facilities in Roseville and Fort Collins to handle some high-end products and other areas of testing.
One of the last things we watched was basic grounding checks – a spark-generator around the casing to try to get the system to fail. Again, simple test, but really cool watching the sparks jump.
Our final visit was to the Reliability lab, where we got to look at a number of tests looking at failures, as well as material analysis and life testing.
John Olivaris walked us through the materials labs, which are used to look at products and components either in development or after failure. They have a number of tools at their disposal including a FT-IR lab with microscopy, material identification tools and libraries, and an XRF (X-Ray Fluorescence) which is used for metal layer thickness analysis.
The lab also has a cross-sectioning microscope for failure analysis as well as a SEM (scanning electron microscope) for resolutions beyond 10,000X magnification. SEM requires a coated sample, making it more difficult to look at metallized surfaces. The lab has a large number of thermal cycling chambers that Michael Fahrenbach showed us, to accelerate any potential temperature-based failures, and also have instrumented many of the tests to observe conduction and other properties in real-time as the tests are conducted.
Henry Garcia showed off what was my second favorite thing – a real-time X-Ray test. With this they are able to put a full laptop display on the system, scan around, and zoom in real-time. He had a display with a broken connection embedded deep in the layers that would have been very difficult to see any other way. With this system information is gathered and fed back to engineering to determine what happened and how to deal with it in the future. Really cool stuff.
Other tests they perform in the lab are HALT – highly accelerated life testing, as well as an Instron for normal force displacement testing to see how products perform under normal life forces as well as more aggressive shocks and bending forces.
In the Reliability Engineering Lab, John Vijil took us through some similar tests we had seen before but done here for engineering failure analysis rather than for compliance testing. They have shaker tables for hard drives, and a room for electrostatic discharge, where it took 10000V to shut down system – which then immediately restarted (in other words, it wasn’t fried!)
There was another room for small-scale reliability testing of interface components. There was a multi-purpose robot that performs tablet-centric tests like presses, swipes and other gestures. There was also a laptop hinge open-close tester that worked on several laptops simultaneously (for context of why this is important, Google “Nintendo DS hinge crack” or “HP 200 hinge crack”).
They also had a screen press test for laptop touchscreens, a keyboard and mouse test that rubs and looks for print wear, a displacement test for keys, and a touchpad abrasion test that puts the pad through 250,000 cycles! The lab has a number of environmental test chambers, such as a dust chamber and a thermal shock chamber that can cycle from 40C to -40C in 1 second!
These labs are all testing to standards, and work together on failure analysis to do teardowns and corrosion analysis, microscopy, and so on.
Oh, and of course they really took care of us in terms of food as well!
I return to my initial question – WHY would HP want us in their labs? Simple – they are representative of both the history and the leading edge of computer testing and reliability, they have stayed true to their ideals even as the market has shifted and margins have thinned, and they have a commitment to quality and reliability that makes me happy that my company has a HP laptop on my desk at work everyday.