Partial Reconfiguration Across FPGAs Steve Wichman1 Sammit Adyha2, Scott Ahrens2, Rohan Ambli2, Brad Alcorn2 Dr. Daniel Connors3, Daniel Fay3 1

Redefine Technologies

2

Colorado Space Grant Consortium

3

University of Colorado – Electrical and Computer Engineering Dept.

 The FPGA redundancy analysis being performed by Redefine Technologies, along with researchers at the University of Colorado, uses three different redundancy methods to decrease the susceptibility of a spacecraft, on a mission survivability level, to electronic failures anywhere throughout the spacecraft. This analysis is a NASA Phase I STTR called Triple3 Redundant Spacecraft Subsystems (T3RSS). By using Field Programmable Gate Array (FPGA) chips, we have analyzed the spacecraftwide benefits of: 

Triplicating the logic and RAM onboard each FPGA using Xilinx's Triple Modular Redundancy (TMR) Tool;



Triplicating the persistent memory storage and hardware access onboard each subsystem; and,



Triplicating the choice of location to run logic (a.k.a. partial reconfiguration across FPGAs), so that subsystem code can run on alternate processors if any component is rendered inoperable

Wichman

Page 1

due to an electronic failure (radiation, manufacturing, human-error, etc). These three methods of triplication should significantly increase the reliability of non-radiation hardened designs, which could allow non-radiation hardened, commercial off-the-shelf (COTS) processing components to be considered as flight critical hardware. The analysis that was performed predicts an increased benefit to any future spacecraft whether it uses radiation hardened (rad-hard) components or not. The T3RSS concept has an opportunity to fly on-board a Colorado Space Grant student-designed mission about the 2008 timeframe to prove the design. The concept of partial reconfiguration across FPGAs proves to be a significant factor that allows designers to maintain 100% mission survivability even using nonrad-hard COTS components. The demonstration being provided to NASA Ames Research Center shows that other components in a spacecraft that were designed to do the tasks for their own subsystems can provide the backup capabilities required should a failure render any other component unusable. A 'code

MAPLD 2006/1029

migration’ method ensures that all essential tasks will be moved from a non-functioning FPGA to another subsystem's FPGA without a loss of mission and at a significant cost savings both in design and testing time. Eliminating dedicated backup components will even reduce weight and volume in some instances on future spacecraft. The T3RSS project is still under investigation. This report constitutes an interim status.   Improved reliability is an obvious thrust of much of the research around FPGAs. More reliable products give any mission a better chance of accomplishing all of its goals. Redundancy offers that reliability by giving the spacecraft electronics a ‘backup’ location in case the first location fails. Triple redundancy is a concept that allocates a third ‘backup’ location. T3RSS attempts to combine three methods of doing triple redundancy – further increasing any specific mission’s survivability level. Using non-radiation hardened (non-radhard) FPGAs in space applications will save money and reduce development time. Although the majority of civil, commercial and military spacecraft require rad-hard components that offer a measurable guarantee that the primary subsystems will work for many years, there are several instances of missions that could use nonrad-hard components. 

University projects – where they cannot afford rad-hard chips.



Responsive Space – where logic development times of rad-hard chips are much longer than on development boards.



NASA or military formation flying missions – where putting multiple rad-hard chips on hundreds of Wichman

Page 2

‘interchangeable’ satellites is too cost prohibitive. 

NASA short-lived missions – where the life expectancy is short enough or the radiation susceptibility is low enough (i.e. reentry or terrestrial vehicles) to warrant non-rad-hard chips.

The T3RSS research being performed, while increasing the chance of non-rad-hard parts flying in space as primary components, should also help identify new design approaches to use on both rad-hard and non-rad-hard projects. The increased level of redundancy provided by T3RSS is applicable to all missions.   The researchers at Redefine Technologies and the University of Colorado are helping to standardize various FPGA redundancy approaches in order to decrease the susceptibility of a spacecraft, on a mission survivability level, to electronic failures anywhere throughout the spacecraft. This analysis is a NASA Phase I STTR called Triple3 Redundant Spacecraft Subsystems (T3RSS). Many engineers have researched susceptibility of specific aspects of Field Programmable Gate Array (FPGA) chips. The failure analysis of spacecraft circuit designs and processor functionality has historically been approached from a ‘per component’ point of view. A plethora of research has been done to validate radiation effects on a particular component. The effects of latch-ups, transcients, and single- and multiple event upsets are increasingly better understood and mitigated very well in the industry. There has even been research on the cascading effects of errors on other devices in a system. However, there has not been a high-level look at maintaining the ability of a MAPLD 2006/1029

system, such as a spacecraft, to perform all of its mission objectives given one or multiple failures on-board. The affect on the missions’ success level have only been implied as engineers continue to analyze at the component level. Since the focus to date has been to avoid the failure in the first place, this analysis looks at a ‘spacecraftwide redundancy plan’ where one or multiple failures of electronic components can still result in near-100% mission success. The T3RSS work establishes the baseline research required to maximize mission survivability even if there are failures in flight.   There are several methods of implementing redundancy within and around FPGAs. The most well known is Triple Modular Redundancy (TMR). TMR is the process of having three independent logic paths all performing the same calculation. At the end of the calculation, the results are compared. Any mis-calculations or bad data can be identified and tossed out. Implementing a well thought out TMR plan within a design helps ensure the right data gets to the right place. However, there may be circumstances where the FPGA, no matter how much TMR is implemented, may not be operational. A few examples include: reaching maximum total ionizing dose levels, loose connectors after launch; singlepoint of failures between logic and actuators; and possibly even data corruption once the signals leave the FPGA. Using non-rad-hard commercial off-the-shelf (COTS) components significantly increase these chances of a component not being able to accurately do its function either temporarily or permanently. Another redundancy method to avoid logic failures is to periodically scrub the FPGA’s configuration. This method of rewriting the logic on the chip several times per second will help ‘erase’ errors in the Wichman

Page 3

logic paths radiation.

that

were

introduced

by

Partial reconfiguration (PR) is the ability to reconstruct a portion of the FPGA while the remainder of the design is still intact1. Typically, PR is used to replace a portion of logic to provide slightly different functionality of the device (such as changing the frequency of a software defined radio) or to scrub just a portion of the device without shutting down the entire device. As a redundancy method however, this means that the logic within an area of an FPGA can ‘moved’ to a different place on the chip, or to a different chip altogether. Moving logic to a different component essentially uses the other existing FPGAs as backup circuitry.   T3RSS is a project that will analyze the spacecraft-wide benefits of: 

Triplicating the logic and RAM onboard each FPGA using Xilinx's Triple Modular Redundancy (TMR) Tool;



Triplicating the persistent memory storage and hardware access onboard each subsystem; and,



Triplicating the choice of location to run logic (a.k.a. partial reconfiguration across FPGAs), so that subsystem code can run on alternate processors if any component is rendered inoperable due to an electronic failure (radiation, manufacturing, human-error, etc).

The T3RSS project is creating a series of demonstrations that incrementally implement the redundancy methods available. Starting with a simple scrubbing 1

Xilinx, XAPP 290: Two Flows for Partial Reconfiguration: Module Based or Difference Based, September 9, 2004. MAPLD 2006/1029

algorithm, the demonstrations build in all three of the redundancy methods discussed while maintaining access to hardware and memory needed throughout the mission. We carefully considered many factors when it came to designing redundant spacecraft subsystems and we came up with several rules we had to follow: 

The system had to be ‘stand-alone’. In space, satellite electronics need to be fully autonomous and able to operate without human intervention. Therefore, the T3RSS architecture would have to ‘heal’ itself from any detected anomaly.



Do not add more single point failures when implementing redundancy. The key to avoiding mission failures is to manage single-point failures by implementing backups or redesigning the system to eliminate single-point failures altogether.



We should not add weight. In spacecraft design, extra weight just increases cost; therefore, T3RSS should denote all weight increases.

We baselined the Xilinx Virtex II Pro (V2P) chip – specifically a XV2P30. This chip was on the Xilinx University Program (XUP) developmental board and provided a ‘medium-sized’ FPGA with two embedded PowerPC (PPC) 405 processors.    The partial reconfiguration redundancy method we investigated involved splitting up an FPGA into regions. These regions house the functionality needed at a particular time during the life of a satellite. Each individual section of logic that can be placed in a region is known as a partial reconfigurable module, or PRM. As the satellite progresses through its lifecycle, the functionality needed Wichman

Page 4

changes, and the logic cores in each region may change. Also, as failures are encountered in a particular region or throughout the entire FPGA, the functionality in that region or FPGA can be switched over to a different region or a different FPGA altogether. When choosing a method to perform the partial reconfiguration, two options were available to implement for a stand-alone system. We could either select a separate controller board that would program the FPGAs through their JTAG ports, or we could use the HWICAP LogiCore which programs the chip via the Internal Configuration Access Port (ICAP) pins. We tested the HWICAP core when implementing our scrubbing algorithm. The scrubbing demo created, simply overwrote the configuration frames of the FPGA periodically (on the order of two times per second). The HWICAP module allows us to read single configuration frames of the FPGA into Block RAM, compare that configuration to the original bitstream stored in memory, identify any changes, and then write the original bitstream (one frame at a time) back to the board, thus ‘erasing’ any changes that may have been introduced due to radiation (see Figure 1). The HWICAP demo we started with read and replaced full bitstreams and had to avoid overwriting LUT RAMs and LUT shiftregisters because of the Virtex II architecture. Using HWICAP on PRMs and its implication on LUT RAMs and LUT shift registers are discussed under the Design Choices section of this paper. The initial HWICAP demo, however, did illustrate one of the primary redundancy methods that can be standardized on any mission design. We chose to create a Bitstream Controller Board (BCB) for our first implementation of a PR controller. A block diagram of the BCB and child FPGAs is shown in Figure 2. MAPLD 2006/1029

The Flash memory on the BCB contains PRMs for a motor controller and an audio controller. Each functionality had two PRM versions, an A and a B. These versions contained the same logic, but they were compiled for the A and B regions of the chip. As discussed later in this paper, the ability to have one placed and routed bitstream be able to ‘fit’ into both regions is not available yet, therefore, the two must exist in order to offer an alternate place to run the logic on the same chip. However, the same PRM compiled for region A on FPGA #1, can, and has been demonstrated to run on region A of FPGA #2.

Figure 1: HWICAP demo.

Figure 2: BCB configuration to program PR regions.

The BCB demo successfully illustrated the ability to run logic in any one of four places within two FPGAs. However, using the BCB in flight is not good practice because the BCB itself is a single-point failure – if it goes down, then we loose our backup ability to move reconfigurable modules. The BCB can be TMRed to reduce the likelihood of bad data, but if the entire BCB chip goes down, then we cannot reprogram our flight FPGAs. Our next step, therefore, is to move the bitstream controller functionality to the FPGA itself and control the PR regions through the HWICAP core. This planned demonstration is shown in Figure 3.   We looked at several design choices to maximize mission survivability. 

Figure 3: ICAP configuration of PR regions.

Wichman

Page 5



When using PR as a redundancy method, we specifically analyzed what the physical layout of the chip should look like. This analysis wanted to be as generic as possible so that it did not exclude any of the various chips in production due to their physical differences such as embedded processors or the amount of BRAM and MAPLD 2006/1029

1)

A

B

2)

D A

C

B F

D

3)

E C G

4)

C A

E D

B

C A

E

B

D

Figure 4: Different options to layout partial reconfiguration regions on an FPGA.

IOBs. This would allow the standardized approach to still apply no matter what chip was chosen. The V2P chip chosen has two PowerPC (PPC) cores embedded in the logic fabric. There were many PR region layout options considered as shown in Figure 4. Because we were working with a V2P that had two processors embedded in it, our analysis had to work around them without making them ‘necessary’ in our design. Table 1 shows the resources available for each region suggested in Figure 4. Region within layout

Layout option

CLBs in BRAMs in region region 736 30 736 30 736 30 736 30 480 16

All of the regions in Figure 4 need to remain slightly variable in size to accommodate routing between IOBs and PR regions, and for bus logic between PR regions and other components. Figure 5 illustrates one example of this concept.

IOBs adjacent 139 139 139 139

1a b c d unused

x region 2 to 32 34 to 64 2 to 32 34 to 64 2 to 64

y region 2 to 37 2 to 37 58 to 93 59 to 93 39 to 55

2a b c d e f g

2 to 10 23 to 43 56 to 64 11 to 22 44 to 55 11 to 22 44 to 55

2 to 93 2 to 93 2 to 93 2 to 37 2 to 37 22 to 93 58 to 93

560 1280 560 256 256 256 256

20 36 20 15 15 15 15

188 64 188 16 16 16 16

3a b c d e

2 to 10 56 to 64 2 to 64 2 to 64 23 to 43

39 to 55 39 to 55 2 to 37 58 to 93 39 to 55

112 112 1472 1472 256

4 4 60 60 8

32 32 220 220 0

4a b c d e

2 to 10 56 to 64 11 to 55 11 to 55 23 to 43

2 to 93 2 to 93 2 to 37 58 to 93 39 to 55

560 560 1024 1024 256

20 20 44 44 8

188 188 64 64 0

Table 1: Layout option breakdown by region.

The T3RSS research is not trying to choose a specific layout for all missions. The layout that is best suited for a particular mission could be any one of the ones above, however, we chose to investigate Wichman

option 4 because it offered a balance of large PR regions (over 1000 CLBs) and medium PR regions (over 500 CLBs). In addition, the fifth region, region E, was a small region that just fit the PR Configuration Manager IP used in later demonstrations.

Page 6

Figure 5: An example of the routing needed between IOBs and PR regions.

There was one pitfall to choose option 4 in Figure 4, however. That pitfall is still under investigation. Technically, whenever we replace region c during our scrubbing routine, the logic in region d is automatically scrubbed because FPGA configuration logic is written in columnar frames that extend the

MAPLD 2006/1029

entire height of the FPGA. According to the Xilinx AppNote on HWICAP2: Although a single CLB LUT or flip-flop can be modified, the underlying mechanism requires that the full column be read into Block RAM. This implies that other logic in the same column can be modified. In most cases, this effect can be ignored. When the frame is written back to the configuration memory the sections of the column that were not modified are written with the same data. Because the FPGA memory cells have glitchless transitions, when rewritten, the unmodified logic will continue to operate unaffected. Two exceptions to this rule exist: when LUTs are configured in Shift Register Mode or as a RAM. If a LUT is modified or just read back in a column that also has a LUT RAM or LUT shift register, then the LUT or shift register will be interrupted and it will lose its state. To resolve the problem, the LUT shift registers and LUT RAMs should be placed in columns that are not read back or modified. If the LUT RAMs or shift register in a column do not change state during the readback or modification, then they will maintain their state. If HWICAP is used to download partial bitstreams from the mainstream tools then there are issues with LUT RAM or LUT shift register placement or operation since the partial bitstreams configure full columns of the FPGA. The T3RSS demonstration was planning on using the HWICAP to download partial bitstreams into regions A through D. The issue is that region C is directly over region D and rewriting only region C would have significant consequences on the LUTs in region D. New Xilinx designs in the Virtex 4 and Virtex 5 are moving these LUTs to

2

Xilinx Logicore OPB HWICAP, March 2004, DS 280, Product Specification, p. 69-70 Wichman

Page 7

‘protected space’ so that scrubbing does not affect them, however, our Virtex II Pro chips do not have this capability. 



When a PRM is placed and routed, the logic paths are established for a particular region. The absolute addresses of the CLBs are set in the bitstream. There is new technology coming soon from Xilinx that will make these addresses ‘relative’ so that intra-FPGA bitstreams can be compiled and the PRM for one region can be run in any region of the same size and shape, and with the same resources as the original region. That technology was not available at the time of this research, but alternatives do exist in the mean time. If there is enough room to store multiple images of the same logic, then multiple bitstreams can be generated for alternative locations for a particular PRM. For instance, in Figure 3, the PRM for the motor controller has bitstreams for regions A and B. The logic that decides which PRM to load can decide which region to place with new code should a higher priority code need to go in either region A or region B.    Logic should be separated by priority and broken into the smallest possible standalone blocks. This allows a higher priority block of code to replace a lower block in order to maximize mission survivability. The PRMs should also be classified as to the mission phase they are needed in. Many satellites have different phases of their life where all functionality is not needed. For instance, during on-orbit maneuvering, most high resolution cameras are not active because the vibration and changing angles of the satellite would not result in usable images. During this ‘maneuver phase’, any logic cores that are not needed then drop priority and the logic MAPLD 2006/1029

cores that control the thrusters and actuators increase their priority. This means that thruster logic cores can now use the space that an imager logic core used during this temporary phase of the mission. When the thrusting is completed, the satellite changes modes back to its ‘science mode’ where the imager priority is higher than the thrusters, and those logic cores replace the valve actuator logic cores. Table 2 shows a table of typical satellite subsystems and the logic needed to function properly. Figure 6 shows what the FPGAs might look like at the different phases. Subsystem

PRM/IP

Encoding Communication Parsing Cmd

Compression Imager Antenna Steering Valve Actuating Propulsion Orbit Insertion

Mission Phase

Priority (1=high)

Transmitting Maneuvering Science Transmitting Maneuvering Science Transmitting Maneuvering Science Transmitting Maneuvering Science Transmitting Maneuvering Imaging Transmitting Maneuvering Imaging

1 2 2 1 5 5 10 10 1 5 10 1 2 1 10 n/a 1 n/a

Table 2: Example logic core classification and prioritization.



loaded. The PR Configuration Manager, or PRCM, is an integral piece of the FPGA redundancy and needs to be redundant. Typical TMR was applied. Since each FPGA needs too have one PRCM running, this particular IP cannot be moved between FPGAs. Additionally, an FPGA needs an PRCM in order to load itself into alternate regions on the same chip, therefore, the PRCM cannot be moved within the same FPGA either, and thus forms an exception to our redundancy model. The PRCM was also chosen to reside in logic because not all spacecraft designs will have embedded PPC processors on all the FPGAs on-board to handle the ICAP operations.

Transmitting Phase

Maneuvering Phase

Science Phase



Choosing when and where to load logic will be handled by a dedicated piece of logic or code. The PR Configuration Manager IP we are developing has a list of spacecraft PRMs, their priorities, and their mission phases as illustrated in Table 2. A bus networked between all the FPGAs on the satellite transports messages between the chips that let each Configuration Manager module know which PRMs are currently Wichman

Page 8

Figure 6: FPGA configurations at different mission phases.

As the mission phase changes during normal operations, or as errors are encountered, then the PRCM will load a new set of PRMs. Some of the PRMs may be the same, in which case the logic is essentially scrubbed. The PRCM signals to MAPLD 2006/1029

mission phase Transmitting

Maneuvering

Science

# hex dec

PRM priority default load list #1 #2 priority default load list #1 #2 priority default load list #1 #2

1 000000000001 … 1 1

2

3 …

4 …

5 …

6 …

7 …

8 …

9 …

10 …

11 …

12 …

2 1

4 1

8 1

16 2

32 1

64 1

128 1

256 10

512 10

1024 10

2048 10

000000001111 000011110000 1

1

10

1

5

1

1

1

10

10

6

5

100000001011 010011100000 10

10

1

1

5

5

1

1

2

5

2

5

101000001100 010101100000

orig load list

#1 #2

000000001111 000011110000

new load list

#1 #2

00000000x111 000011101000

Given: error in region a (where PRM 4 was running) Given: Transmitting

Notes: for each mission phase, there is a new mask to run against

Table 3: Example of generating a new load list.

the other FPGAs in the system which ones it successfully loaded. Careful consideration has been given to not load a PRM that has already been loaded by another FPGA. For instance, the default configuration of any particular FPGA is established at launch. Before the mission, each FPGA knows the PRMs it will load first. During the mission, this configuration can change. The default configuration is masked against the current configuration of all FPGAs, and a new PRM load list is generated. Table 3 shows an example of how a load list is generated. After the given error in the region where PRM 4 was running, FPGA #1 called for a ‘re-allocation’. The ‘re-allocation’ resulted in FPGA #2 running the higher priority PRM 4, and the lower priority PRM 5 was chosen not to run. The PRCM keeps track of which PRMs are currently running, the priority of each PRM, and the fully operational regions.    The more tightly you initially pack an FPGA at launch, the fewer options you have to move around higher priority blocks. If a second FPGA has a smaller utilization at the time of a shift, then the logic core that Wichman

Page 9

did not have room after the shift within FPGA #1 would still be able to run on the alternate chip. Figures 7 & 8 illustrate how 50-75% utilized FPGAs can accommodate PRMs from another failed FPGA.   Over the next few months, we will finish investigating several key issues.    As you move PR blocks within and between FPGAs, you have to ensure that the hardware and memory components needed by that logic are still accessible. SpaceWire networking components are being investigated for their redundancy properties. A mass memory component and all hardware can ‘hang’ off the SpaceWire bus and be accessed from any FPGA in the system. The multiple router design of the bus allows data path redundancy that follows the T3RSS philosophy. 



As PRMs are moved between FPGAs, or PRMs are moved within the same FPGA, the block memory being utilized at the time of the failure could contain important MAPLD 2006/1029

information that will need to be transferred to the new PR region. It is anticipated that the Xilinx BRAM scrubber can be modified to serve this need. The BRAM scrubber is a replacement macro that actually triplicates the BRAM output and contains a scrubber controller3. It is this scrubber controller than could be modified to save a ‘copy’ of the ‘voted’ contents to a global memory location during its scrubbing duties.

   An analysis of the capabilities and the configuration of each FPGA will be investigated. If the clocks differ between PR regions, there should be enough clock managers to support new regions that are introduced to the FPGA. Many of the functions needed on-board a satellite require a processor to perform some calculations that would be too prohibitive in logic. Any functions performed within the processor would have to retain their ability to talk to hardware and memory located on elsewhere on the board. If memory is shifted around, or the logic core needed to support the functions on the processor are moved, the architecture needs to be able to handle that.   

Figure 7: Spacecraft subsystem processing before a failure.

If PR regions are of the same priority and have the same duty cycle (at a different time phase), then the FPGA can timemultiplex these regions and implement them both. Examples of time-multiplexed PRMs will be analyzed as a redundancy option.   

Figure 8: Spacecraft subsystem processing after a failure.

As PRMs are moved around, their paths get longer or shorter. The timing between memory and hardware is changed. The design must keep these in mind and the designer must run timing constraints for the best and worst case scenarios as found in any of the FPGAs that it could be moved to. A formal methodology will be developed to standardize the design analysis that is needed to implement the T3RSS approach.

3

SEU Mitigation of a Soft Embedded Processor in the Virtex-II FPGAs, Rezgui, MAPLD 2005/E238, www.klabs.org/mapld05/presento/238_rezgui_p.ppt Wichman

Page 10

MAPLD 2006/1029

 

other IPs can run in their place if resources are limited.

The downside of implementing the T3RSS redundancy approach (and the downside of implementing any redundancy approach) is an increased complexity. More research on the T3RSS methodology may standardize some of the redundancy approaches outlined here and ease the tradeoff decisions that may be needed between an increased complexity and increased mission survivability.   This research is still underway but at first glance, it seems that using partial reconfiguration as a redundancy method may offer alternate design techniques that can increase mission survivability. The following topics will help deciphering the results of this research.    As mentioned in the paper, there are several characteristics of spacecraft logic that can be combined into formal design documentation. Similar to an Interface Control Document (ICD), a Partial Reconfiguration Options (PRO) Document can be compiled to let other designers know how to ‘accommodate’ your logic on their FPGA should a failure occur on your platform and vice versa. This PRO Document can be a collaborative, satellitewide, systems-level document that will ‘grow’ to outline all the PRMs on the satellite. The PRO Document will contain data such as in the example shown in Figure 9. This document will facilitate selecting a PR region layout, possibly identify alternate regions an IP can run in, and determine when PRMs should be loaded and when they can be ‘stored’ so

Wichman

Page 11

Figure 9: PRO Document example

   A high level analysis will determine the best method of gauging mission success. Figure 10 shows a few analyses that may help designers express the anticipated outcome of their designs. The future work outlined above should help flush the benefits of these gauges.

Figure 10: Mission survivability analysis.

MAPLD 2006/1029