# Benchmarking

## Evaluation of Electromagnetic Software

Revised: June 4, 1997

© Copyright 1994, 1999 Sonnet Software, Inc. All Rights Reserved

### Table of Contents

Chapter 1 - Introduction

Chapter 2 - Technical Overview

Chapter 3 - The Stripline Standard

Chapter 4 - The Stripline Standard and Triangular Subsections

Chapter 5 - The Microstrip Standard

Chapter 6 - The Coupled Microstrip Standard

Chapter 7 - Limit Tests

Chapter 8 - Lossy Ground Plane and Lossy Dielectric Tests

## Chapter 1 - Introduction

GABMAC | Error Definition | Philosophy | Error and Accuracy | What Error Level Is Needed?

At Sonnet, we started working on microwave electromagnetic software in 1983. By 1989, we considered the software ready for a production release to the general microwave design community. Since then, Sonnet has rapidly climbed to dominance in the market niche we describe as "3-D Planar Electromagnetic Microwave Software". We did this by providing the fastest, most accurate electromagnetic software available anywhere in the world. We wrote the software not only with serious attention to the underlying complex electromagnetic theory, but also with significant effort devoted to the two additional areas of software engineering (software testing, porting, maintenance, user interface, etc.) and microwave engineering (obtaining and providing data in the format that the microwave engineer finds most useful).

But new customers, especially engineers, are not convinced by high sounding claims such as those above. As the saying goes, "In God we trust, all others must have proof." In achieving our high level of success, we have had to provide proof at every step of every sale to every customer, one at a time.

In this document, we assume that you are considering the purchase of electromagnetic software. Our objective is to make you as informed as possible about the whole process, so that you can decide exactly what it is that you want and whether or not a given product is a match with your needs, whatever electromagnetic product that may be.

It is common to evaluate an electromagnetic tool by applying it to an in-house circuit, say a filter or an amplifier. We highly recommend this approach in conjunction with the benchmarks provided in this document, however, the engineer should be aware of the limitations of such an approach.

For example, when the validation is complete, you have a measured versus calculated plot. Normally we look at the agreement between measured and calculated data and announce that we have "Good Agreement Between Measured And Calculated", so-called GABMAC. Everyone walks away with a nice warm, fuzzy feeling.

But what we really need to look at is the *difference* between measured and
calculated. This is the error. Now, how much of the error is due to measurement and how
much is due to analysis error? If the circuit is complicated, how much is due to that
capacitor and how much is due to this resistor, etc.? Also, in a large, complex circuit,
errors tend to cancel (Root Sum Squared, RSS error is what is seen). While desirable for a
given design, it does not allow us to quantitatively evaluate the intrinsic underlying
error of the analysis, which is the objective of the evaluation. Warm, fuzzy feelings are
important too, but we are engineers. We need numbers.

If we are going to measure error quantitatively, we need to define it quantitatively.

To calculate percent error, take the difference between the calculated value and the correct value, divide by the correct value and multiply by 100:

Percent Error = 100 × ( *Calculated* - *Correct* ) / *Correct*

Note that percent error of S-Parameters is not useful, since the correct value of an S-Parameter may be zero (as it is in the stripline standard to follow) and division by zero is undefined.

We have found that, for any given structure, it is best to define error in terms of an underlying lumped model. For example if we loosely state that the S-Parameters of a step discontinuity have 5% error, what we really mean is that the series inductance and shunt capacitances of the equivalent underlying lumped model are in error by up to 5%.

If we are dealing with a transmission line, then the equivalent infinite cascade of series L - shunt C are in error by up to 5%. Or, more simply, the velocity of propagation and characteristic impedance are each in error by up to 5%.

No single benchmark can test all accuracy issues. In addition no set of benchmarks can substitute for the confidence obtained from evaluating in-house designed and measured circuits. The set of benchmarks that we provide here (and will be updating on a continuous basis) are selected for 1) Simplicity, and 2) Accuracy. In other words, the benchmarks must be simple to capture and the correct results must be known as accurately as possible, preferably with exact results available.

Simplicity allows the benchmarks to be quickly used on a wide variety of analyses. Exact results allows us to eliminate measurement error. Thus, we may obtain noise-free results on the error of an analysis and use this as a basis for comparison.

One difficulty with simple benchmarks is that they do not test for circuit complexity. Of course, in a properly performing electromagnetic analysis, accuracy does not depend on circuit simplicity or complexity but only on subsection or mesh size. However, it would be nice to have quantitative confirmation of this. Unfortunately, there are no complex circuits for which we have exact results. Thus, testing for circuit complexity is left to the GABMAC evaluation of existing in-house circuitry, and is outside the scope of this document.

In addition, as mentioned earlier, an appropriate comparison compares analysis speed at a given level of error. It is inappropriate to compare the speed of two analyses when one is being operated at a much higher level of accuracy. Thus, the result of a benchmark is a plot of analysis time versus analysis error. Because the quantities can cover a wide range, we use a log-log plot. When evaluations are performed using the same computer, such an analysis time versus error plot is appropriate to use in a direct quantitative comparison between two competing electromagnetic tools.

Sometimes the terms "error" and "accuracy" are used interchangeably. In reality, error is the opposite of accuracy. Small error means high accuracy. If an analysis has a 5% error, the accuracy is 95%.

A common (puffing) statement is, "Analysis accuracy is on the order of measurement error." Note that this statements actually means the opposite of what is intended.

It turns out that most electromagnetic analyses can achieve 5% to 10% error quite easily. Also, coincidentally, this is about the same error level that a pure circuit theory analysis can achieve on a modestly compacted circuit design. Thus, if a 5% to 10% error level is acceptable (i.e., circuit theory gives a reasonable answer), by all means, use circuit theory, don't bother with the time and expense of electromagnetics.

The more common situation is that circuit theory works moderately well for the initial design, but not quite well enough to achieve the highly desired success on first fabrication. In this case you need to invoke electromagnetics on the tightly compacted portions of the circuit, the parts of the circuit where circuit theory has problems. We have found, through the experience of our customers, that a consistent level of about 1% error in the critical portions of a circuit is required to realize a high probability of success on first fabrication. When the error level creeps up to 5% in important areas of a circuit, it can severely impact the probability of success on first fabrication.

Note that a 5% error level can sneak through an un-critical GABMAC (see above) validation. Such an error level has no chance of going undetected in the benchmarks which follow.

For these reasons, we generally recommend that when electromagnetics is invoked, critical portions of the circuit should be analyzed as close to the 1% error level (i.e., use small subsection size) as possible. This does mean a slower analysis time, however, it increases the probability of success on first fabrication dramatically, a very worthwhile trade-off.

On very rare occasions, accuracy on the order of 0.1% is needed. Electromagnetic analyses which can achieve this level are very rare, and any analyses used at this level need to be carefully and rigorously checked for error performance.