Contents | June 2008 | Volume 3, Issue 2
Consolidation and Transformation Gaining Momentum
After months of careful planning and preparation, agencies participating in the Texas data center services program are accelerating consolidation and transformation activities. Transformation is the implementation of upgraded software, tools, and processes, and consolidation is the migration from older hardware in legacy agency data centers to fewer, newer machines in the Austin and San Angelo Data Centers. The two processes go hand in hand: transformation upgrades pave the way for hardware consolidation and further transformation is possible once equipment is consolidated.
With 27 participating agencies and at least as many different ways of doing business, agencies, DIR, and Team for Texas face technical, organizational, and logistical challenges planning for transformation and consolidation. Plans must consider agency business schedules, current and upcoming projects, the age of hardware, and software versions, among other things. Additionally, plans must be coordinated across the enterprise to maximize efficiency and resource utilization. Initial planning efforts began last year. In 2007, the Texas State Library and Archives Commission (TSLAC) became the first agency to be consolidated.
The lessons learned from the TSLAC consolidation helped agencies, DIR, and Team for Texas improve the planning process. Changes included a renewed focus on pre-planning, development of a standardized approach to consolidation, and the creation of a master schedule for all agency consolidations. The standard approach includes in-depth analysis of server and application dependencies (known as deep dives), development of test and cutover plans, and dry runs before the actual move. While activities differ for server, mainframe, and print/mail moves, all tasks are planned in detail, with the actual migration described in an hour-by-hour plan.
The individual agency consolidation plans form the Texas Data Center Services Integrated Consolidation Schedule. This master schedule summarizes all agency consolidation plans and interdependencies. DIR references this schedule to coordinate consolidation activities or make changes to accommodate changing business needs.
Recent months have seen an increase in the number of agencies consolidating service delivery to the data center. Consolidations to date include the server migration for DIR; mainframe moves for the Texas Workforce Commission (TWC) and Texas Youth Commission (TYC); and print/mail consolidations for the Health and Human Services Commission, Texas Department of Agriculture, Railroad Commission of Texas,and Texas Higher Education Coordinating Board. This edition of the Data Center Download highlights a few transformation/consolidation projects completed recently and the role of mail in service delivery.
Detailed Plans and Agency Involvement Make Difference in TWC Migration
Over Memorial Day weekend, when most people were enjoying time off with family and friends, IT staff from TWC, DIR, and Team for Texas were busy migrating applications on the TWC mainframe and 14 closely coupled servers to new equipment in the Austin Data Center. This weekend of intense activity capped months of detailed preparation that enabled a successful and smooth transition.
The TWC mainframe and closely coupled servers host several applications that support the agency’s programs, including Unemployment Insurance (UI), UI employment wage record data, UI employer taxes, a workforce development case management system, and the popular job matching website WorkInTexas.com. Planning efforts centered around minimizing down time and ensuring future availability of these important public services.
All participants agree that the most important contributor to the success of the migration was commitment from all parties. Both TWC and Team for Texas dedicated staff; including a project manager, mainframe lead, network lead, print/mail lead, and server lead, to plan and manage the effort. Many people throughout the TWC business areas helped test the applications before and during the migration weekend. TWC also clearly defined requirements for the move including cutover times, outage windows, and testing plans. This professional approach helped Team for Texas understand expectations and develop an approach that met the agency’s needs.
All Texas DCS mainframe migrations include three dry runs (phased practice tests) before the actual cutover from old to new equipment. In the first dry run, Team for Texas staff review equipment configurations and test hardware and software functions. The second dry run involves both the agency and Team for Texas and consists of backup, restoration, and testing of applications. The team identifies and attempts to resolve all technical issues uncovered by the exercise. The third dry run is a true dress rehearsal and should meet all requirements as identified in the plan: it must demonstrate that the teams have resolved any issues from dry run two and prove that they are ready for the actual migration. This careful and deliberate planning, where each phase builds on the previous one, helps the team uncover and fix unexpected issues before they affect production.
The team managed the migration weekend with the same attention to detail and care as the preplanning. There were status updates every four hours and a conference call every 12 hours to keep all parties informed about progress. Team for Texas started at 5:00 a.m. Saturday by backing up the mainframe systems to tape. Then they transported the tapes to the Austin Data Center, loaded them, booted up the mainframe, and completed a series of checks for the IPL (initial program load). The server data was transmitted over the data circuit between TWC and the Austin Data Center. Team for Texas turned over the system to TWC for extensive application testing and at 5:00 p.m. Sunday there was a “go or no go” decision. If there had been any indication of a potential problem, the team could have returned service to the old mainframe and servers. This was not necessary and the entire process was successfully completed, within the designated 48 hour schedule. After the go decision was made, the production network was routed to the equipment at the Austin Data Center and tested before opening up to TWC’s external customers.
Looking back on the migration, TWC data processing director Robert VonQuintus acknowledges the work effort. To other agencies, he says, “I want to let them know it can be done. You can be successful. It takes work and strong agency involvement, but we did it.”
Phase by Phase Approach Facilitates DIR Server Migration
Unlike a mainframe migration, a server migration can be broken into subprojects to reduce risk and to organize activities around agency business operations. The DIR server migration, which included 25 servers, was divided into four subprojects. Each subproject included preparation of the infrastructure, server and data migration, user acceptance testing (similar to the dry runs for the mainframe migration), and the actual cutover. Each subproject was managed independently, enabling the teams from DIR and Team for Texas to develop realistic timelines and spread the work over several weeks.
Breaking the migration into multiple pieces also enabled the teams apply lessons learned to the next subproject. During the first subproject, the team uncovered issues with the migration tool and firewall configurations. Additionally, DIR had some unconventional operating system configurations necessitating software patches. While these issues slowed progress of the first subproject, they were corrected early on. Future subprojects were executed without incident and ahead of schedule.
Because the subprojects consisted of just a few servers each, they were completed during the week after business hours. DIR’s IT staff worked closely with Team for Texas to minimize the impact to end users. DIR staff were asked to log off the system on migration nights. Connections were automatically rerouted to the new servers upon logging in the next morning. Although relatively small, DIR’s move did include technology upgrades and server virtualization (to enable the 10 new servers to operate as the 25 old servers did). These changes increased the complexity of the project and required DIR staff to make changes to several internal applications.
As both end users and liaisons to the service provider, DIR’s Technology Center Operations Division (TCO) had a unique vantage point. TCO Division Director Lara Coffer commented, “I thought things went very smoothly. I know how hard staff worked to make this happen and as an end user, I truly appreciated it.”
Mail Services: Reliably Delivered
Mail services play a vital role in DCS agencies’ communications with consumers and businesses. Each month, the Pitney Bowes mail team processes over 4.2 million pieces of correspondence: more than one piece for every 100 Texans. The Health and Human Services Commission (HHSC) and the Texas Workforce Commission (TWC) process the most mail: 3.1 million pieces and 1 million pieces, respectively, including TIERS eligibility letters, invoices for HHSC program services, and TWC appeals packets. The mail team also handles smaller jobs for the Railroad Commission of Texas (permit renewals), Texas Department of Agriculture (letters and invoices), and Texas Higher Education Coordinating Board (letters and notices). Most of these items are processed at the Austin Data Center, with some mail scheduled to be migrated from the legacy agency facilities in the next several months.
HHSC’s medical IDs, which account for approximately 2.4 million pieces every month, are the largest single job at the Austin mail center. Medical IDs function as proof of insurance for CHIP, Medicare, and Medicaid participants. On each visit to a doctor, hospital, pharmacy, or other medical provider, participants must show the medical ID to prove current eligibility and ensure proper payment for the services. The ID job is so large it takes 100 hours to print and seven calendar days for the mail team to complete document folding, stuffing, and metering of all the envelopes. Because the IDs must be mailed by a certain date each month, the team works around the clock and over the weekend, as necessary, to get the job done.
High tech equipment in use at the Austin Data Center has improved tracking of medical IDs and several other mail jobs. Each ID is printed with a unique bar code that is verified more than 15 times during mail processing. The extensive tracking enables operators to know where every single piece of mail is at all times, identify if any pieces have jammed, and reprint only damaged pieces as necessary. The machine automatically stops if bar codes are out of sequence, facilitating quick diagnosis and correction of the misfeed. Over time, bar codes will be added to more mail jobs, so similar tracking can take place for other types of correspondence.
Since the first mail services migrated to the data center in November, performance has been steadily improving as operators learn the new equipment and work processes. Management has identified the root causes of problems and taken steps to fix the issue including process changes and additional quality checks to make sure mail is processed efficiently and accurately.
Debbie Peña, Pitney Bowes area operations director, has seen a notable increase in teamwork. She says, “I’m so proud of our people. Everyone is working together and taking ownership of the process.” Employees have even suggested process improvements. This employee commitment yields results: from January through May 2008, the team achieved 100 percent success on critical jobs (over 18 million pieces including the Medical IDs) and 97.7 percent success with standard jobs (like non-sensitive letters and notices). And, in April and May, the team exceeded a “six sigma” level of quality. Six sigma is a disciplined approach used by companies like General Electric and Motorola for process improvement. The goal of six sigma is 3.4 defects or less per million opportunities. Surpassing six sigma means with more than 8.2 million pieces processed in April and May, there were just 23 items with any kind of a defect. The mail team is taking steps to improve this even further.
