|
It's easy to make mistakes when testing software or planning a testing effort. Some mistakes are made so often, so repeatedly, by so many different people, that they deserve the label Classic Mistake.
在測(cè)試軟件或制訂測(cè)試工作計(jì)劃時(shí)很容易犯一些錯(cuò)誤。有些錯(cuò)誤經(jīng)常被許多不同的人一而再、再而三地犯，應(yīng)該被列為典型錯(cuò)誤。
Classic mistakes cluster usefully into five groups, which I've called "themes":
典型錯(cuò)誤可以有效地分為五組，我把這些組稱為“主題”。
· The Role of Testing: who does the testing team serve, and how does it do that?
· 測(cè)試的作用：誰(shuí)承擔(dān)測(cè)試小組的責(zé)任，如何做？
· Planning the Testing Effort: how should the whole team's work be organized?
· 制訂測(cè)試工作計(jì)劃：應(yīng)該如何組織整個(gè)小組的工作？
· Personnel Issues: who should test?
· 人員問題：誰(shuí)應(yīng)該做測(cè)試？
· The Tester at Work: designing, writing, and maintaining individual tests.
· 工作中的測(cè)試員：設(shè)計(jì)、編寫和維護(hù)各測(cè)試。
· Technology Rampant: quick technological fixes for hard problems.
· 過度使用技術(shù)：艱難問題的快速技術(shù)修復(fù)
I have two goals for this paper. First, it should identify the mistakes, put them in context, describe why they're mistakes, and suggest alternatives. Because the context of one mistake is usually prior mistakes, the paper is written in a narrative style rather than as a list that can be read in any order. Second, the paper should be a handy checklist of mistakes. For that reason, the classic mistakes are printed in a larger bold font when they appear in the text, and they're also summarized at the end.
本文有兩個(gè)目標(biāo)。第一，應(yīng)當(dāng)識(shí)別錯(cuò)誤，將它們放到具體環(huán)境中，描述它們?yōu)槭裁词清e(cuò)誤，并給出替代方法的建議。因?yàn)橐粋�(gè)錯(cuò)誤的具體環(huán)境通常是先決錯(cuò)誤，所以本文將以敘事的方式而不是以可以按任意順序閱讀的列表方式來描述。第二，本文應(yīng)該是一個(gè)便于查看的錯(cuò)誤列表。因?yàn)檫@個(gè)原因，文章中出現(xiàn)的典型錯(cuò)誤都以大號(hào)粗體字印刷，并在文章的結(jié)尾處匯總。
Although many of these mistakes apply to all types of software projects, my specific focus is the testing of commercial software products, not custom software or software that is safety critical or mission critical.
雖然這些錯(cuò)誤很多都適用于所有類型的軟件項(xiàng)目，但我的重點(diǎn)將放在商用軟件產(chǎn)品的測(cè)試上，而不是定制軟件或者是高度安全或關(guān)鍵任務(wù)的軟件測(cè)試上。
This paper is essentially a series of bug reports for the testing process. You may think some of them are features, not bugs. You may disagree with the severities I assign. You may want more information to help in debugging, or want to volunteer information of your own. Any decent bug reporting system will treat the original bug report as the first part of a conversation. So should it be with this paper. Therefore, follow this link for an ongoing discussion of this topic.
本文主要是測(cè)試過程的一系列錯(cuò)誤報(bào)告。你可能認(rèn)為它們中的部分屬于特性問題而不是 bug。你可能不贊成我設(shè)定的嚴(yán)重性級(jí)別。你可能需要更多的信息以用于幫助排除錯(cuò)誤，或者希望提供你自己的信息。任何設(shè)計(jì)良好的錯(cuò)誤報(bào)告系統(tǒng)都將原始的錯(cuò)誤報(bào)告當(dāng)作是對(duì)話的起始部分。本文也是這樣，所以，可以按照鏈接參加這個(gè)主題的討論。
Theme One: The Role of Testing
主題一：測(cè)試的作用
A first major mistake people make is thinking that the testing team is responsible for assuring quality. This role, often assigned to the first testing team in an organization, makes it the last defense, the barrier between the development team (accused of producing bad quality) and the customer (who must be protected from them). It's characterized by a testing team (often called the "Quality Assurance Group") that has formal authority to prevent shipment of the product. That in itself is a disheartening task: the testing team can't improve quality, only enforce a minimal level. Worse, that authority is usually more apparent than real. Discovering that, together with the perverse incentives of telling developers that quality is someone else's job, leads to testing teams and testers who are disillusioned, cynical, and view themselves as victims. We've learned from Deming and others that products are better and cheaper to produce when everyone, at every stage in development, is responsible for the quality of their work ([Deming86], [Ishikawa85]).
人們犯的第一個(gè)主要錯(cuò)誤是認(rèn)為測(cè)試小組應(yīng)當(dāng)負(fù)責(zé)質(zhì)量保證。這個(gè)角色常常分配給組織中的第一測(cè)試小組，將它作為最后的防御，成為開發(fā)小組（被指責(zé)為產(chǎn)生低劣質(zhì)量）和客戶（必須受到保護(hù)以遠(yuǎn)離低劣質(zhì)量）的一個(gè)屏障。它的特征是測(cè)試小組（常稱為“質(zhì)量保證組”）表面上具有阻止產(chǎn)品發(fā)貨的權(quán)力。 這本身是一個(gè)令人沮喪的任務(wù)：測(cè)試小組不能提高質(zhì)量，只能強(qiáng)制一個(gè)最低水平。更糟糕的是，這種權(quán)力常常是看上去比實(shí)際的重要。如果發(fā)現(xiàn)這一點(diǎn)，再加上有違常理地暗示開發(fā)人員質(zhì)量是別人的事情，導(dǎo)致測(cè)試小組和測(cè)試員感到失望、憤事嫉俗、感覺自己是受害者。我們從Deming 和其他人的工作可以得知：如果每個(gè)人都在開發(fā)的各個(gè)階段對(duì)他們的工作質(zhì)量負(fù)責(zé)，則產(chǎn)品會(huì)又好又便宜（[Deming86]，[Ishikawa85]）。
In practice, whatever the formal role, most organizations believe that the purpose of testing is to find bugs. This is a less pernicious definition than the previous one, but it's missing a key word. When I talk to programmers and development managers about testers, one key sentence keeps coming up: "Testers aren't finding the important bugs." Sometimes that's just griping, sometimes it's because the programmers have a skewed sense of what's important, but I regret to say that all too often it's valid criticism. Too many bug reports from testers are minor or irrelevant, and too many important bugs are missed.
實(shí)際上，不管表面上的作用是什么，大多數(shù)組織都相信測(cè)試的目的是發(fā)現(xiàn) bug。這個(gè)定義的危害比前一個(gè)定義的危害要小，但是忽略了一個(gè)關(guān)鍵詞。當(dāng)我同程序員和開發(fā)經(jīng)理談到測(cè)試員的時(shí)候，不時(shí)聽到一個(gè)關(guān)鍵的句子：測(cè)試員找不到重要的 bug。有時(shí)候這種說法只是一種抱怨，有時(shí)候是因?yàn)槌绦騿T對(duì)于什么是正確的感覺不對(duì)，但我很遺憾地說，它們經(jīng)常是有效的批評(píng)。測(cè)試員的太多的bug 報(bào)告是微小的、不相關(guān)的，而有太多重要的錯(cuò)誤都被遺漏了。
What's an important bug? Important to whom? To a first approximation, the answer must be "to customers". Almost everyone will nod their head upon hearing this definition, but do they mean it? Here's a test of your organization's maturity. Suppose your product is a system that accepts email requests for service. As soon as a request is received, it sends a reply that says "your request of 5/12/97 was accepted and its reference ID is NIC-051297-3". A tester who sends in many requests per day finds she has difficulty keeping track of which request goes with which ID. She wishes that the original request were appended to the acknowledgement. Furthermore, she realizes that some customers will also generate many requests per day, so would also appreciate this feature. Would she:
什么是重要的 bug？對(duì)誰(shuí)而言是重要的？直觀的估計(jì)，答案肯定是“對(duì)于客戶”。聽到這個(gè)定義，幾乎每個(gè)人都會(huì)點(diǎn)頭稱是，但他們確實(shí)這樣認(rèn)為嗎？這里要測(cè)試一下你們組織的成熟度。假設(shè)你們的產(chǎn)品是一個(gè)接受電子郵件請(qǐng)求服務(wù)的系統(tǒng)。當(dāng)收到請(qǐng)求時(shí)，它馬上發(fā)送一個(gè)“您在97年5月12日發(fā)送的請(qǐng)求已經(jīng)受理，參考ID是NIC-051297-3”的答復(fù)。一個(gè)每天發(fā)送很多請(qǐng)求的測(cè)試員發(fā)現(xiàn)要分清楚哪個(gè)請(qǐng)求與哪個(gè)ID對(duì)應(yīng)是非常困難的。她希望最初的請(qǐng)求能夠附加在確認(rèn)郵件的后面。并且，她意識(shí)到某些可戶可能每天也會(huì)產(chǎn)生很多請(qǐng)求，所以會(huì)高度評(píng)價(jià)這個(gè)功能的。那么她將：
1. file a bug report documenting a usability problem, with the expectation that it will be assigned a reasonably high priority (because the fix is clearly useful to everyone, important to some users, and easy to do)?
寫一個(gè) bug 報(bào)告，記錄一個(gè)可用性問題，希望能夠分配一個(gè)合理的高優(yōu)先級(jí)（因?yàn)檫@個(gè)修復(fù)很明顯對(duì)每個(gè)人都很用，對(duì)有部分用戶來說還非常重要，并且也容易修改）？
2. file a bug report with the expectation that it will be assigned "enhancement request" priority and disappear forever into the bug database?
寫一個(gè) bug 報(bào)告，希望它被分配為“功能提升請(qǐng)求”優(yōu)先級(jí)并永遠(yuǎn)從 bug 數(shù)據(jù)庫(kù)中消失？
3. file a bug report that yields a "works as designed" resolution code, perhaps with an email "nastygram" from a programmer or the development manager?
寫一個(gè) bug 報(bào)告，產(chǎn)生一個(gè)“按設(shè)計(jì)工作”解決碼，可能還加上一個(gè)來自程序員或開發(fā)經(jīng)理的“不同意”電子郵件?
4. not bother with a bug report because it would end up in cases (2) or (3)?
不打算費(fèi)事去寫 bug 報(bào)告，因?yàn)樗鼘⒁郧闆r(2)或(3)結(jié)束？
If usability problems are not considered valid bugs, your project defines the testing task too narrowly. Testers are restricted to checking whether the product does what was intended, not whether what was intended is useful. Customers do not care about the distinction, and testers shouldn't either.
如果可用性問題不認(rèn)為是有效的 bug，那么你們的項(xiàng)目將測(cè)試任務(wù)定義得太狹窄了。測(cè)試員嚴(yán)格限制為檢查產(chǎn)品是否按預(yù)期工作，而不管這種預(yù)期是否有效。客戶不關(guān)心這個(gè)區(qū)別，測(cè)試員也不應(yīng)該關(guān)心。
Testers are often the only people in the organization who use the system as heavily as an expert. They notice usability problems that experts will see. (Formal usability testing almost invariably concentrates on novice users.) Expert customers often don't report usability problems, because they've been trained to know it's not worth their time. Instead, they wait (in vain, perhaps) for a more usable product and switch to it. Testers can prevent that lost revenue.
測(cè)試員經(jīng)常是組織中唯一像專家一樣大量使用系統(tǒng)的人。他們會(huì)注意到專家會(huì)看到的可用性問題。（形式上的可用性測(cè)試幾乎不可避免地集中于沒有經(jīng)驗(yàn)的用戶。）專家客戶常常不會(huì)報(bào)告可用性問題，因?yàn)樗麄円呀?jīng)被訓(xùn)練的知道不值得花時(shí)間去這樣做。相反，他們（也許是徒勞地）等待下一個(gè)可用的產(chǎn)品然后切換過去。測(cè)試員可以避免這個(gè)損失。
While defining the purpose of testing as "finding bugs important to customers" is a step forward, it's more restrictive than I like. It means that there is no focus on an estimate of quality (and on the quality of that estimate). Consider these two situations for a product with five subsystems.
將測(cè)試的目的定義為“找到對(duì)用戶重要的 bug ”是向前進(jìn)了一步，但與我所喜歡定義相比仍有限制。這意味著沒有集中于質(zhì)量評(píng)估（以及這種評(píng)估的質(zhì)量）。考慮一下測(cè)試含有五個(gè)子系統(tǒng)的產(chǎn)品的兩種情況。
1. 100 bugs are found in subsystem 1 before release. (For simplicity, assume that all bugs are of the highest priority.) No bugs are found in the other subsystems. After release, no bugs are reported in subsystem 1, but 12 bugs are found in each of the other subsystems.
在發(fā)布前，在子系統(tǒng)1中找到了100個(gè)bug 。（為了簡(jiǎn)單起見，假設(shè)所有的 bug 都是最高級(jí)別的。）在其他子系統(tǒng)中沒有發(fā)現(xiàn) bug 。在發(fā)布后，在子系統(tǒng)1中沒有報(bào)告 bug ，但在其他每個(gè)子系統(tǒng)中都報(bào)告了12個(gè) bug 。
2. Before release, 50 bugs are found in subsystem 1. 6 bugs are found in each of the other subsystems. After release, 50 bugs are found in subsystem 1 and 6 bugs in each of the other subsystems.
在發(fā)布前，在子系統(tǒng)1中找到了50個(gè) bug 。在其他每個(gè)子系統(tǒng)中都找到了6個(gè) bug 。在發(fā)布后，在子系統(tǒng)1中報(bào)告了50個(gè) bug ，在其他每個(gè)子系統(tǒng)中都報(bào)告了6個(gè) bug。
From the "find important bugs" standpoint, the first testing effort was superior. It found 100 bugs before release, whereas the second found only 74. But I think you can make a strong case that the second effort is more useful in practical terms. Let me restate the two situations in terms of what a test manager might say before release:
從“找到重要 bug”的觀點(diǎn)看，第1種測(cè)試情況較為理想。在發(fā)布前找到了100個(gè) bug ，但是第2種情況中只找到74個(gè)。但我想你們可能會(huì)提出一個(gè)有力的理由認(rèn)為第2中測(cè)試在實(shí)際中更有用。讓我以產(chǎn)品發(fā)版前測(cè)試經(jīng)理可能說些什么來重新描述一下兩種測(cè)試情況：
1. "We have tested subsystem 1 very thoroughly, and we believe we've found almost all of the priority 1 bugs. Unfortunately, we don't know anything about the bugginess of the remaining five subsystems."
“我們?nèi)鏈y(cè)試了子系統(tǒng)1，我們相信已經(jīng)找出了幾乎所有優(yōu)先級(jí)為1的 bug。不幸的是，我們對(duì)其他五個(gè)子系統(tǒng)的的 bug 一無(wú)所知。”
2. "We've tested all subsystems moderately thoroughly. Subsystem 1 is still very buggy. The other subsystems are about 1/10th as buggy, though we're sure bugs remain."
“我們比較全面地測(cè)試了所有的子系統(tǒng)。子系統(tǒng)1仍舊有不少 bug。其他子系統(tǒng)雖然還有 bug，但只有子系統(tǒng)1的 bug 的十分之一。”
This is, admittedly, an extreme example, but it demonstrates an important point. The project manager has a tough decision: would it be better to hold on to the product for more work, or should it be shipped now? Many factors - all rough estimates of possible futures - have to be weighed: Will a competitor beat us to release and tie up the market? Will dropping an unfinished feature to make it into a particular magazine's special "Java Development Environments" issue cause us to suffer in the review? Will critical customer X be more annoyed by a schedule slip or by a shaky product? Will the product be buggy enough that profits will be eaten up by support costs or, worse, a recall?
必須承認(rèn)，這是一個(gè)極端的例子，但是證明了一個(gè)重要的觀點(diǎn)。項(xiàng)目經(jīng)理有一個(gè)艱難的決定：是延遲產(chǎn)品交付，再工作一段時(shí)間，還是現(xiàn)在就交付使用？許多因素——都是一些大致的評(píng)估——都必須予以權(quán)衡：競(jìng)爭(zhēng)對(duì)手會(huì)搶先發(fā)布產(chǎn)品并占領(lǐng)市場(chǎng)嗎？如果丟掉一個(gè)未完工的功能部件會(huì)使得某一個(gè)雜志的 “Java 開發(fā)環(huán)境” 特別期刊的評(píng)論中對(duì)我們?cè)斐蓳p害嗎？關(guān)鍵客戶 X 對(duì)產(chǎn)品延期和劣質(zhì)產(chǎn)品哪一個(gè)更感到煩惱？產(chǎn)品是否有很多 bug，以至于支持成本會(huì)吃掉利潤(rùn)，或者更糟糕的是將產(chǎn)品召回？
The testing team will serve the project manager better if it concentrates first on providing estimates of product bugginess (reducing uncertainty), then on finding more of the bugs that are estimated to be there. That affects test planning, the topic of the next theme.
如果測(cè)試小組首先集中于產(chǎn)品錯(cuò)誤的估計(jì)（減少不確定性），然后再找到更多的錯(cuò)誤，他們會(huì)更好地服務(wù)于項(xiàng)目經(jīng)理。這會(huì)影響測(cè)試計(jì)劃。測(cè)試計(jì)劃將在下個(gè)主題中論述。
It also affects status reporting. Test managers often err by reporting bug data without putting it into context. Without context, project management tends to focus on one graph:
這也會(huì)影響狀態(tài)報(bào)告。測(cè)試經(jīng)理常常會(huì)被沒有放到具體環(huán)境中的報(bào)告 bug數(shù)據(jù)誤導(dǎo)。沒有具體環(huán)境，項(xiàng)目管理傾向于集中于一幅圖：
The flattening in the curve of bugs found will be interpreted in the most optimistic possible way unless you as test manager explain the limitations of the data:
平滑的錯(cuò)誤曲線很容易以一種樂觀的方式解釋，除非你作為測(cè)試經(jīng)理解釋了數(shù)據(jù)的局限：
· "Only half the planned testing tasks have been finished, so little is known about half the areas in the project. There could soon be a big spike in the number of bugs found."
· 只有一半的計(jì)劃測(cè)試做完了，對(duì)于項(xiàng)目的一半所知甚少。很快就有很多錯(cuò)誤要被發(fā)現(xiàn)了。
· "That's especially likely because the last two weekly builds have been lightly tested. I told the testers to take their vacations now, before the project hits crunch mode."
· 很有可能這樣，因?yàn)樵谶^去的兩個(gè)周構(gòu)建只是略微測(cè)試了一下。我告訴測(cè)試員在項(xiàng)目進(jìn)入艱難狀態(tài)之前，現(xiàn)在開始休假。
· "Furthermore, based on previous projects with similar amounts and kinds of testing effort, it's reasonable to expect at least 45 priority-1 bugs remain undiscovered. Historically, that's pretty high for a successful product."
· 并且，根據(jù)以前的經(jīng)驗(yàn)，可以預(yù)料到至少還有45個(gè)級(jí)別為1的 bug還沒有發(fā)現(xiàn)。從歷史看，這對(duì)于一個(gè)成功產(chǎn)品來說是很高的。
For discussions of using bug data, see [Cusumano95], [Rothman96], and [Marick97].
關(guān)于使用 bug 數(shù)據(jù)的討論，請(qǐng)參閱[Cusumano95]、[Rothman96]和[Marick97]。
Earlier I asserted that testers can't directly improve quality; they can only measure it. That's true only if you find yourself starting testing too late. Tests designed before coding begins can improve quality. They inform the developer of the kinds of tests that will be run, including the special cases that will be checked. The developer can use that information while thinking about the design, during design inspections, and in his own developer testing.
我在前面說過，測(cè)試員不能直接提高質(zhì)量，他們只能評(píng)估它。只有在你發(fā)現(xiàn)測(cè)試開始得太晚的時(shí)候，這種說法才是正確的。在編碼開始前設(shè)計(jì)測(cè)試將會(huì)提高質(zhì)量。他們讓開發(fā)人員知道將進(jìn)行什么樣的測(cè)試，將檢查哪些特殊用例。開發(fā)人員在思考設(shè)計(jì)、審查設(shè)計(jì)和自己做測(cè)試的時(shí)候可以使用這些信息。
Early test design can do more than prevent coding bugs. As will be discussed in the next theme, many tests will represent user tasks. The process of designing them can find user interface and usability problems before expensive rework is required. I've found problems like no user-visible place for error messages to go, pluggable modules that didn't fit together, two screens that had to be used together but could not be displayed simultaneously, and "obvious" functions that couldn't be performed. Test design fits nicely into any usability engineering effort ([Nielsen93]) as a way of finding specification bugs.
盡早測(cè)試的作用不僅僅是防止編碼錯(cuò)誤。像我們將在下一個(gè)主題中所討論的那樣，許多測(cè)試代表的是用戶任務(wù)。設(shè)計(jì)它們的過程可以在昂貴的重新工作之前發(fā)現(xiàn)用戶界面和可用性問題。我發(fā)現(xiàn)過的問題包括：錯(cuò)誤消息不能顯示在用戶可以看到的地方，插件不能放到一起，兩個(gè)必須同時(shí)使用的屏幕不能同時(shí)顯示，一個(gè)“很明顯”的功能不能執(zhí)行。測(cè)試設(shè)計(jì)作為一個(gè)發(fā)現(xiàn)規(guī)格說明書 bug 的方法，很好地與可用性工程工作相適應(yīng)([Nielsen93])。
I should note that involving testing early feels unnatural to many programmers and development managers. There may be feelings that you are intruding on their turf or not giving them the chance to make the mistakes that are an essential part of design. Take care, especially at first, not to increase their workload or slow them down. It may take one or two entire projects to establish your credibility and usefulness.
我應(yīng)當(dāng)說明早期介入測(cè)試對(duì)于許多程序員和開發(fā)經(jīng)理來說不自然�？赡苡幸环N感覺是你干擾了他們，沒有給他們?cè)谠O(shè)計(jì)的基礎(chǔ)部分犯錯(cuò)誤的機(jī)會(huì)。小心些，尤其是在開始的時(shí)候，不要增加他們的工作量或減慢了他們的速度�？赡苄枰恢羶蓚�(gè)完整的項(xiàng)目才能建立你們的可信度并顯示出作用。
內(nèi)容太多，共31頁(yè)。后面的見附件。
掃碼關(guān)注5G通信官方公眾號(hào),免費(fèi)領(lǐng)取以下5G精品資料
1、回復(fù)“YD5GAI”免費(fèi)領(lǐng)取《中國(guó)移動(dòng)：5G網(wǎng)絡(luò)AI應(yīng)用典型場(chǎng)景技術(shù)解決方案白皮書》
2、回復(fù)“5G6G”免費(fèi)領(lǐng)取《5G_6G毫米波測(cè)試技術(shù)白皮書-2022_03-21》
3、回復(fù)“YD6G”免費(fèi)領(lǐng)取《中國(guó)移動(dòng)：6G至簡(jiǎn)無(wú)線接入網(wǎng)白皮書》
4、回復(fù)“LTBPS”免費(fèi)領(lǐng)取《《中國(guó)聯(lián)通5G終端白皮書》》
5、回復(fù)“ZGDX”免費(fèi)領(lǐng)取《中國(guó)電信5G NTN技術(shù)白皮書》
6、回復(fù)“TXSB”免費(fèi)領(lǐng)取《通信設(shè)備安裝工程施工工藝圖解》
7、回復(fù)“YDSL”免費(fèi)領(lǐng)取《中國(guó)移動(dòng)算力并網(wǎng)白皮書》
8、回復(fù)“5GX3”免費(fèi)領(lǐng)取《 R16 23501-g60 5G的系統(tǒng)架構(gòu)1》
作者的更多帖子
|
|
|
|
|
| |
|
關(guān)注樓主