esy-bash

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"> <head> <meta http-equiv="Content-Type" content="application/xhtml+xml; charset=UTF-8" /> <meta name="generator" content="AsciiDoc 8.6.9" /> <title>Fighting regressions with git bisect</title> <style type="text/css"> /* Shared CSS for AsciiDoc xhtml11 and html5 backends */ /* Default font. */ body { font-family: Georgia,serif; } /* Title font. */ h1, h2, h3, h4, h5, h6, div.title, caption.title, thead, p.table.header, #toctitle, #author, #revnumber, #revdate, #revremark, #footer { font-family: Arial,Helvetica,sans-serif; } body { margin: 1em 5% 1em 5%; } a { color: blue; text-decoration: underline; } a:visited { color: fuchsia; } em { font-style: italic; color: navy; } strong { font-weight: bold; color: #083194; } h1, h2, h3, h4, h5, h6 { color: #527bbd; margin-top: 1.2em; margin-bottom: 0.5em; line-height: 1.3; } h1, h2, h3 { border-bottom: 2px solid silver; } h2 { padding-top: 0.5em; } h3 { float: left; } h3 + * { clear: left; } h5 { font-size: 1.0em; } div.sectionbody { margin-left: 0; } hr { border: 1px solid silver; } p { margin-top: 0.5em; margin-bottom: 0.5em; } ul, ol, li > p { margin-top: 0; } ul > li { color: #aaa; } ul > li > * { color: black; } .monospaced, code, pre { font-family: "Courier New", Courier, monospace; font-size: inherit; color: navy; padding: 0; margin: 0; } pre { white-space: pre-wrap; } #author { color: #527bbd; font-weight: bold; font-size: 1.1em; } #email { } #revnumber, #revdate, #revremark { } #footer { font-size: small; border-top: 2px solid silver; padding-top: 0.5em; margin-top: 4.0em; } #footer-text { float: left; padding-bottom: 0.5em; } #footer-badges { float: right; padding-bottom: 0.5em; } #preamble { margin-top: 1.5em; margin-bottom: 1.5em; } div.imageblock, div.exampleblock, div.verseblock, div.quoteblock, div.literalblock, div.listingblock, div.sidebarblock, div.admonitionblock { margin-top: 1.0em; margin-bottom: 1.5em; } div.admonitionblock { margin-top: 2.0em; margin-bottom: 2.0em; margin-right: 10%; color: #606060; } div.content { /* Block element content. */ padding: 0; } /* Block element titles. */ div.title, caption.title { color: #527bbd; font-weight: bold; text-align: left; margin-top: 1.0em; margin-bottom: 0.5em; } div.title + * { margin-top: 0; } td div.title:first-child { margin-top: 0.0em; } div.content div.title:first-child { margin-top: 0.0em; } div.content + div.title { margin-top: 0.0em; } div.sidebarblock > div.content { background: #ffffee; border: 1px solid #dddddd; border-left: 4px solid #f0f0f0; padding: 0.5em; } div.listingblock > div.content { border: 1px solid #dddddd; border-left: 5px solid #f0f0f0; background: #f8f8f8; padding: 0.5em; } div.quoteblock, div.verseblock { padding-left: 1.0em; margin-left: 1.0em; margin-right: 10%; border-left: 5px solid #f0f0f0; color: #888; } div.quoteblock > div.attribution { padding-top: 0.5em; text-align: right; } div.verseblock > pre.content { font-family: inherit; font-size: inherit; } div.verseblock > div.attribution { padding-top: 0.75em; text-align: left; } /* DEPRECATED: Pre version 8.2.7 verse style literal block. */ div.verseblock + div.attribution { text-align: left; } div.admonitionblock .icon { vertical-align: top; font-size: 1.1em; font-weight: bold; text-decoration: underline; color: #527bbd; padding-right: 0.5em; } div.admonitionblock td.content { padding-left: 0.5em; border-left: 3px solid #dddddd; } div.exampleblock > div.content { border-left: 3px solid #dddddd; padding-left: 0.5em; } div.imageblock div.content { padding-left: 0; } span.image img { border-style: none; vertical-align: text-bottom; } a.image:visited { color: white; } dl { margin-top: 0.8em; margin-bottom: 0.8em; } dt { margin-top: 0.5em; margin-bottom: 0; font-style: normal; color: navy; } dd > *:first-child { margin-top: 0.1em; } ul, ol { list-style-position: outside; } ol.arabic { list-style-type: decimal; } ol.loweralpha { list-style-type: lower-alpha; } ol.upperalpha { list-style-type: upper-alpha; } ol.lowerroman { list-style-type: lower-roman; } ol.upperroman { list-style-type: upper-roman; } div.compact ul, div.compact ol, div.compact p, div.compact p, div.compact div, div.compact div { margin-top: 0.1em; margin-bottom: 0.1em; } tfoot { font-weight: bold; } td > div.verse { white-space: pre; } div.hdlist { margin-top: 0.8em; margin-bottom: 0.8em; } div.hdlist tr { padding-bottom: 15px; } dt.hdlist1.strong, td.hdlist1.strong { font-weight: bold; } td.hdlist1 { vertical-align: top; font-style: normal; padding-right: 0.8em; color: navy; } td.hdlist2 { vertical-align: top; } div.hdlist.compact tr { margin: 0; padding-bottom: 0; } .comment { background: yellow; } .footnote, .footnoteref { font-size: 0.8em; } span.footnote, span.footnoteref { vertical-align: super; } #footnotes { margin: 20px 0 20px 0; padding: 7px 0 0 0; } #footnotes div.footnote { margin: 0 0 5px 0; } #footnotes hr { border: none; border-top: 1px solid silver; height: 1px; text-align: left; margin-left: 0; width: 20%; min-width: 100px; } div.colist td { padding-right: 0.5em; padding-bottom: 0.3em; vertical-align: top; } div.colist td img { margin-top: 0.3em; } @media print { #footer-badges { display: none; } } #toc { margin-bottom: 2.5em; } #toctitle { color: #527bbd; font-size: 1.1em; font-weight: bold; margin-top: 1.0em; margin-bottom: 0.1em; } div.toclevel0, div.toclevel1, div.toclevel2, div.toclevel3, div.toclevel4 { margin-top: 0; margin-bottom: 0; } div.toclevel2 { margin-left: 2em; font-size: 0.9em; } div.toclevel3 { margin-left: 4em; font-size: 0.9em; } div.toclevel4 { margin-left: 6em; font-size: 0.9em; } span.aqua { color: aqua; } span.black { color: black; } span.blue { color: blue; } span.fuchsia { color: fuchsia; } span.gray { color: gray; } span.green { color: green; } span.lime { color: lime; } span.maroon { color: maroon; } span.navy { color: navy; } span.olive { color: olive; } span.purple { color: purple; } span.red { color: red; } span.silver { color: silver; } span.teal { color: teal; } span.white { color: white; } span.yellow { color: yellow; } span.aqua-background { background: aqua; } span.black-background { background: black; } span.blue-background { background: blue; } span.fuchsia-background { background: fuchsia; } span.gray-background { background: gray; } span.green-background { background: green; } span.lime-background { background: lime; } span.maroon-background { background: maroon; } span.navy-background { background: navy; } span.olive-background { background: olive; } span.purple-background { background: purple; } span.red-background { background: red; } span.silver-background { background: silver; } span.teal-background { background: teal; } span.white-background { background: white; } span.yellow-background { background: yellow; } span.big { font-size: 2em; } span.small { font-size: 0.6em; } span.underline { text-decoration: underline; } span.overline { text-decoration: overline; } span.line-through { text-decoration: line-through; } div.unbreakable { page-break-inside: avoid; } /* * xhtml11 specific * * */ div.tableblock { margin-top: 1.0em; margin-bottom: 1.5em; } div.tableblock > table { border: 3px solid #527bbd; } thead, p.table.header { font-weight: bold; color: #527bbd; } p.table { margin-top: 0; } /* Because the table frame attribute is overriden by CSS in most browsers. */ div.tableblock > table[frame="void"] { border-style: none; } div.tableblock > table[frame="hsides"] { border-left-style: none; border-right-style: none; } div.tableblock > table[frame="vsides"] { border-top-style: none; border-bottom-style: none; } /* * html5 specific * * */ table.tableblock { margin-top: 1.0em; margin-bottom: 1.5em; } thead, p.tableblock.header { font-weight: bold; color: #527bbd; } p.tableblock { margin-top: 0; } table.tableblock { border-width: 3px; border-spacing: 0px; border-style: solid; border-color: #527bbd; border-collapse: collapse; } th.tableblock, td.tableblock { border-width: 1px; padding: 4px; border-style: solid; border-color: #527bbd; } table.tableblock.frame-topbot { border-left-style: hidden; border-right-style: hidden; } table.tableblock.frame-sides { border-top-style: hidden; border-bottom-style: hidden; } table.tableblock.frame-none { border-style: hidden; } th.tableblock.halign-left, td.tableblock.halign-left { text-align: left; } th.tableblock.halign-center, td.tableblock.halign-center { text-align: center; } th.tableblock.halign-right, td.tableblock.halign-right { text-align: right; } th.tableblock.valign-top, td.tableblock.valign-top { vertical-align: top; } th.tableblock.valign-middle, td.tableblock.valign-middle { vertical-align: middle; } th.tableblock.valign-bottom, td.tableblock.valign-bottom { vertical-align: bottom; } /* * manpage specific * * */ body.manpage h1 { padding-top: 0.5em; padding-bottom: 0.5em; border-top: 2px solid silver; border-bottom: 2px solid silver; } body.manpage h2 { border-style: none; } body.manpage div.sectionbody { margin-left: 3em; } @media print { body.manpage div#toc { display: none; } } </style> <script type="text/javascript"> /*<![CDATA[*/ var asciidoc = { // Namespace. ///////////////////////////////////////////////////////////////////// // Table Of Contents generator ///////////////////////////////////////////////////////////////////// /* Author: Mihai Bazon, September 2002 * http://students.infoiasi.ro/~mishoo * * Table Of Content generator * Version: 0.4 * * Feel free to use this script under the terms of the GNU General Public * License, as long as you do not remove or alter this notice. */ /* modified by Troy D. Hanson, September 2006. License: GPL */ /* modified by Stuart Rackham, 2006, 2009. License: GPL */ // toclevels = 1..4. toc: function (toclevels) { function getText(el) { var text = ""; for (var i = el.firstChild; i != null; i = i.nextSibling) { if (i.nodeType == 3 /* Node.TEXT_NODE */) // IE doesn't speak constants. text += i.data; else if (i.firstChild != null) text += getText(i); } return text; } function TocEntry(el, text, toclevel) { this.element = el; this.text = text; this.toclevel = toclevel; } function tocEntries(el, toclevels) { var result = new Array; var re = new RegExp('[hH]([1-'+(toclevels+1)+'])'); // Function that scans the DOM tree for header elements (the DOM2 // nodeIterator API would be a better technique but not supported by all // browsers). var iterate = function (el) { for (var i = el.firstChild; i != null; i = i.nextSibling) { if (i.nodeType == 1 /* Node.ELEMENT_NODE */) { var mo = re.exec(i.tagName); if (mo && (i.getAttribute("class") || i.getAttribute("className")) != "float") { result[result.length] = new TocEntry(i, getText(i), mo[1]-1); } iterate(i); } } } iterate(el); return result; } var toc = document.getElementById("toc"); if (!toc) { return; } // Delete existing TOC entries in case we're reloading the TOC. var tocEntriesToRemove = []; var i; for (i = 0; i < toc.childNodes.length; i++) { var entry = toc.childNodes[i]; if (entry.nodeName.toLowerCase() == 'div' && entry.getAttribute("class") && entry.getAttribute("class").match(/^toclevel/)) tocEntriesToRemove.push(entry); } for (i = 0; i < tocEntriesToRemove.length; i++) { toc.removeChild(tocEntriesToRemove[i]); } // Rebuild TOC entries. var entries = tocEntries(document.getElementById("content"), toclevels); for (var i = 0; i < entries.length; ++i) { var entry = entries[i]; if (entry.element.id == "") entry.element.id = "_toc_" + i; var a = document.createElement("a"); a.href = "#" + entry.element.id; a.appendChild(document.createTextNode(entry.text)); var div = document.createElement("div"); div.appendChild(a); div.className = "toclevel" + entry.toclevel; toc.appendChild(div); } if (entries.length == 0) toc.parentNode.removeChild(toc); }, ///////////////////////////////////////////////////////////////////// // Footnotes generator ///////////////////////////////////////////////////////////////////// /* Based on footnote generation code from: * http://www.brandspankingnew.net/archive/2005/07/format_footnote.html */ footnotes: function () { // Delete existing footnote entries in case we're reloading the footnodes. var i; var noteholder = document.getElementById("footnotes"); if (!noteholder) { return; } var entriesToRemove = []; for (i = 0; i < noteholder.childNodes.length; i++) { var entry = noteholder.childNodes[i]; if (entry.nodeName.toLowerCase() == 'div' && entry.getAttribute("class") == "footnote") entriesToRemove.push(entry); } for (i = 0; i < entriesToRemove.length; i++) { noteholder.removeChild(entriesToRemove[i]); } // Rebuild footnote entries. var cont = document.getElementById("content"); var spans = cont.getElementsByTagName("span"); var refs = {}; var n = 0; for (i=0; i<spans.length; i++) { if (spans[i].className == "footnote") { n++; var note = spans[i].getAttribute("data-note"); if (!note) { // Use [\s\S] in place of . so multi-line matches work. // Because JavaScript has no s (dotall) regex flag. note = spans[i].innerHTML.match(/\s*\[([\s\S]*)]\s*/)[1]; spans[i].innerHTML = "[<a id='_footnoteref_" + n + "' href='#_footnote_" + n + "' title='View footnote' class='footnote'>" + n + "</a>]"; spans[i].setAttribute("data-note", note); } noteholder.innerHTML += "<div class='footnote' id='_footnote_" + n + "'>" + "<a href='#_footnoteref_" + n + "' title='Return to text'>" + n + "</a>. " + note + "</div>"; var id =spans[i].getAttribute("id"); if (id != null) refs["#"+id] = n; } } if (n == 0) noteholder.parentNode.removeChild(noteholder); else { // Process footnoterefs. for (i=0; i<spans.length; i++) { if (spans[i].className == "footnoteref") { var href = spans[i].getElementsByTagName("a")[0].getAttribute("href"); href = href.match(/#.*/)[0]; // Because IE return full URL. n = refs[href]; spans[i].innerHTML = "[<a href='#_footnote_" + n + "' title='View footnote' class='footnote'>" + n + "</a>]"; } } } }, install: function(toclevels) { var timerId; function reinstall() { asciidoc.footnotes(); if (toclevels) { asciidoc.toc(toclevels); } } function reinstallAndRemoveTimer() { clearInterval(timerId); reinstall(); } timerId = setInterval(reinstall, 500); if (document.addEventListener) document.addEventListener("DOMContentLoaded", reinstallAndRemoveTimer, false); else window.onload = reinstallAndRemoveTimer; } } asciidoc.install(); /*]]>*/ </script> </head> <body class="article"> <div id="header"> <h1>Fighting regressions with git bisect</h1> <span id="author">Christian Couder</span><br /> <span id="email"><code><<a href="mailto:chriscool@tuxfamily.org">chriscool@tuxfamily.org</a>></code></span><br /> <span id="revdate">2009/11/08</span> </div> <div id="content"> <div class="sect1"> <h2 id="_abstract">Abstract</h2> <div class="sectionbody"> <div class="paragraph"><p>"git bisect" enables software users and developers to easily find the commit that introduced a regression. We show why it is important to have good tools to fight regressions. We describe how "git bisect" works from the outside and the algorithms it uses inside. Then we explain how to take advantage of "git bisect" to improve current practices. And we discuss how "git bisect" could improve in the future.</p></div> </div> </div> <div class="sect1"> <h2 id="_introduction_to_git_bisect">Introduction to "git bisect"</h2> <div class="sectionbody"> <div class="paragraph"><p>Git is a Distributed Version Control system (DVCS) created by Linus Torvalds and maintained by Junio Hamano.</p></div> <div class="paragraph"><p>In Git like in many other Version Control Systems (VCS), the different states of the data that is managed by the system are called commits. And, as VCS are mostly used to manage software source code, sometimes "interesting" changes of behavior in the software are introduced in some commits.</p></div> <div class="paragraph"><p>In fact people are specially interested in commits that introduce a "bad" behavior, called a bug or a regression. They are interested in these commits because a commit (hopefully) contains a very small set of source code changes. And it’s much easier to understand and properly fix a problem when you only need to check a very small set of changes, than when you don’t know where look in the first place.</p></div> <div class="paragraph"><p>So to help people find commits that introduce a "bad" behavior, the "git bisect" set of commands was invented. And it follows of course that in "git bisect" parlance, commits where the "interesting behavior" is present are called "bad" commits, while other commits are called "good" commits. And a commit that introduce the behavior we are interested in is called a "first bad commit". Note that there could be more than one "first bad commit" in the commit space we are searching.</p></div> <div class="paragraph"><p>So "git bisect" is designed to help find a "first bad commit". And to be as efficient as possible, it tries to perform a binary search.</p></div> </div> </div> <div class="sect1"> <h2 id="_fighting_regressions_overview">Fighting regressions overview</h2> <div class="sectionbody"> <div class="sect2"> <h3 id="_regressions_a_big_problem">Regressions: a big problem</h3> <div class="paragraph"><p>Regressions are a big problem in the software industry. But it’s difficult to put some real numbers behind that claim.</p></div> <div class="paragraph"><p>There are some numbers about bugs in general, like a NIST study in 2002 <a href="#1">[1]</a> that said:</p></div> <div class="quoteblock"> <div class="content"> <div class="paragraph"><p>Software bugs, or errors, are so prevalent and so detrimental that they cost the U.S. economy an estimated $59.5 billion annually, or about 0.6 percent of the gross domestic product, according to a newly released study commissioned by the Department of Commerce’s National Institute of Standards and Technology (NIST). At the national level, over half of the costs are borne by software users and the remainder by software developers/vendors. The study also found that, although all errors cannot be removed, more than a third of these costs, or an estimated $22.2 billion, could be eliminated by an improved testing infrastructure that enables earlier and more effective identification and removal of software defects. These are the savings associated with finding an increased percentage (but not 100 percent) of errors closer to the development stages in which they are introduced. Currently, over half of all errors are not found until "downstream" in the development process or during post-sale software use.</p></div> </div> <div class="attribution"> </div></div> <div class="paragraph"><p>And then:</p></div> <div class="quoteblock"> <div class="content"> <div class="paragraph"><p>Software developers already spend approximately 80 percent of development costs on identifying and correcting defects, and yet few products of any type other than software are shipped with such high levels of errors.</p></div> </div> <div class="attribution"> </div></div> <div class="paragraph"><p>Eventually the conclusion started with:</p></div> <div class="quoteblock"> <div class="content"> <div class="paragraph"><p>The path to higher software quality is significantly improved software testing.</p></div> </div> <div class="attribution"> </div></div> <div class="paragraph"><p>There are other estimates saying that 80% of the cost related to software is about maintenance <a href="#2">[2]</a>.</p></div> <div class="paragraph"><p>Though, according to Wikipedia <a href="#3">[3]</a>:</p></div> <div class="quoteblock"> <div class="content"> <div class="paragraph"><p>A common perception of maintenance is that it is merely fixing bugs. However, studies and surveys over the years have indicated that the majority, over 80%, of the maintenance effort is used for non-corrective actions (Pigosky 1997). This perception is perpetuated by users submitting problem reports that in reality are functionality enhancements to the system.</p></div> </div> <div class="attribution"> </div></div> <div class="paragraph"><p>But we can guess that improving on existing software is very costly because you have to watch out for regressions. At least this would make the above studies consistent among themselves.</p></div> <div class="paragraph"><p>Of course some kind of software is developed, then used during some time without being improved on much, and then finally thrown away. In this case, of course, regressions may not be a big problem. But on the other hand, there is a lot of big software that is continually developed and maintained during years or even tens of years by a lot of people. And as there are often many people who depend (sometimes critically) on such software, regressions are a really big problem.</p></div> <div class="paragraph"><p>One such software is the Linux kernel. And if we look at the Linux kernel, we can see that a lot of time and effort is spent to fight regressions. The release cycle start with a 2 weeks long merge window. Then the first release candidate (rc) version is tagged. And after that about 7 or 8 more rc versions will appear with around one week between each of them, before the final release.</p></div> <div class="paragraph"><p>The time between the first rc release and the final release is supposed to be used to test rc versions and fight bugs and especially regressions. And this time is more than 80% of the release cycle time. But this is not the end of the fight yet, as of course it continues after the release.</p></div> <div class="paragraph"><p>And then this is what Ingo Molnar (a well known Linux kernel developer) says about his use of git bisect:</p></div> <div class="quoteblock"> <div class="content"> <div class="paragraph"><p>I most actively use it during the merge window (when a lot of trees get merged upstream and when the influx of bugs is the highest) - and yes, there have been cases that i used it multiple times a day. My average is roughly once a day.</p></div> </div> <div class="attribution"> </div></div> <div class="paragraph"><p>So regressions are fought all the time by developers, and indeed it is well known that bugs should be fixed as soon as possible, so as soon as they are found. That’s why it is interesting to have good tools for this purpose.</p></div> </div> <div class="sect2"> <h3 id="_other_tools_to_fight_regressions">Other tools to fight regressions</h3> <div class="paragraph"><p>So what are the tools used to fight regressions? They are nearly the same as those used to fight regular bugs. The only specific tools are test suites and tools similar as "git bisect".</p></div> <div class="paragraph"><p>Test suites are very nice. But when they are used alone, they are supposed to be used so that all the tests are checked after each commit. This means that they are not very efficient, because many tests are run for no interesting result, and they suffer from combinational explosion.</p></div> <div class="paragraph"><p>In fact the problem is that big software often has many different configuration options and that each test case should pass for each configuration after each commit. So if you have for each release: N configurations, M commits and T test cases, you should perform:</p></div> <div class="listingblock"> <div class="content"> <pre><code>N * M * T tests</code></pre> </div></div> <div class="paragraph"><p>where N, M and T are all growing with the size your software.</p></div> <div class="paragraph"><p>So very soon it will not be possible to completely test everything.</p></div> <div class="paragraph"><p>And if some bugs slip through your test suite, then you can add a test to your test suite. But if you want to use your new improved test suite to find where the bug slipped in, then you will either have to emulate a bisection process or you will perhaps bluntly test each commit backward starting from the "bad" commit you have which may be very wasteful.</p></div> </div> </div> </div> <div class="sect1"> <h2 id="_git_bisect_overview">"git bisect" overview</h2> <div class="sectionbody"> <div class="sect2"> <h3 id="_starting_a_bisection">Starting a bisection</h3> <div class="paragraph"><p>The first "git bisect" subcommand to use is "git bisect start" to start the search. Then bounds must be set to limit the commit space. This is done usually by giving one "bad" and at least one "good" commit. They can be passed in the initial call to "git bisect start" like this:</p></div> <div class="listingblock"> <div class="content"> <pre><code>$ git bisect start [BAD [GOOD...]]</code></pre> </div></div> <div class="paragraph"><p>or they can be set using:</p></div> <div class="listingblock"> <div class="content"> <pre><code>$ git bisect bad [COMMIT]</code></pre> </div></div> <div class="paragraph"><p>and:</p></div> <div class="listingblock"> <div class="content"> <pre><code>$ git bisect good [COMMIT...]</code></pre> </div></div> <div class="paragraph"><p>where BAD, GOOD and COMMIT are all names that can be resolved to a commit.</p></div> <div class="paragraph"><p>Then "git bisect" will checkout a commit of its choosing and ask the user to test it, like this:</p></div> <div class="listingblock"> <div class="content"> <pre><code>$ git bisect start v2.6.27 v2.6.25 Bisecting: 10928 revisions left to test after this (roughly 14 steps) [2ec65f8b89ea003c27ff7723525a2ee335a2b393] x86: clean up using max_low_pfn on 32-bit</code></pre> </div></div> <div class="paragraph"><p>Note that the example that we will use is really a toy example, we will be looking for the first commit that has a version like "2.6.26-something", that is the commit that has a "SUBLEVEL = 26" line in the top level Makefile. This is a toy example because there are better ways to find this commit with Git than using "git bisect" (for example "git blame" or "git log -S<string>").</p></div> </div> <div class="sect2"> <h3 id="_driving_a_bisection_manually">Driving a bisection manually</h3> <div class="paragraph"><p>At this point there are basically 2 ways to drive the search. It can be driven manually by the user or it can be driven automatically by a script or a command.</p></div> <div class="paragraph"><p>If the user is driving it, then at each step of the search, the user will have to test the current commit and say if it is "good" or "bad" using the "git bisect good" or "git bisect bad" commands respectively that have been described above. For example:</p></div> <div class="listingblock"> <div class="content"> <pre><code>$ git bisect bad Bisecting: 5480 revisions left to test after this (roughly 13 steps) [66c0b394f08fd89236515c1c84485ea712a157be] KVM: kill file->f_count abuse in kvm</code></pre> </div></div> <div class="paragraph"><p>And after a few more steps like that, "git bisect" will eventually find a first bad commit:</p></div> <div class="listingblock"> <div class="content"> <pre><code>$ git bisect bad 2ddcca36c8bcfa251724fe342c8327451988be0d is the first bad commit commit 2ddcca36c8bcfa251724fe342c8327451988be0d Author: Linus Torvalds <torvalds@linux-foundation.org> Date: Sat May 3 11:59:44 2008 -0700 Linux 2.6.26-rc1 :100644 100644 5cf82581... 4492984e... M Makefile</code></pre> </div></div> <div class="paragraph"><p>At this point we can see what the commit does, check it out (if it’s not already checked out) or tinker with it, for example:</p></div> <div class="listingblock"> <div class="content"> <pre><code>$ git show HEAD commit 2ddcca36c8bcfa251724fe342c8327451988be0d Author: Linus Torvalds <torvalds@linux-foundation.org> Date: Sat May 3 11:59:44 2008 -0700 Linux 2.6.26-rc1 diff --git a/Makefile b/Makefile index 5cf8258..4492984 100644 --- a/Makefile +++ b/Makefile @@ -1,7 +1,7 @@ VERSION = 2 PATCHLEVEL = 6 -SUBLEVEL = 25 -EXTRAVERSION = +SUBLEVEL = 26 +EXTRAVERSION = -rc1 NAME = Funky Weasel is Jiggy wit it # *DOCUMENTATION*</code></pre> </div></div> <div class="paragraph"><p>And when we are finished we can use "git bisect reset" to go back to the branch we were in before we started bisecting:</p></div> <div class="listingblock"> <div class="content"> <pre><code>$ git bisect reset Checking out files: 100% (21549/21549), done. Previous HEAD position was 2ddcca3... Linux 2.6.26-rc1 Switched to branch 'master'</code></pre> </div></div> </div> <div class="sect2"> <h3 id="_driving_a_bisection_automatically">Driving a bisection automatically</h3> <div class="paragraph"><p>The other way to drive the bisection process is to tell "git bisect" to launch a script or command at each bisection step to know if the current commit is "good" or "bad". To do that, we use the "git bisect run" command. For example:</p></div> <div class="listingblock"> <div class="content"> <pre><code>$ git bisect start v2.6.27 v2.6.25 Bisecting: 10928 revisions left to test after this (roughly 14 steps) [2ec65f8b89ea003c27ff7723525a2ee335a2b393] x86: clean up using max_low_pfn on 32-bit $ $ git bisect run grep '^SUBLEVEL = 25' Makefile running grep ^SUBLEVEL = 25 Makefile Bisecting: 5480 revisions left to test after this (roughly 13 steps) [66c0b394f08fd89236515c1c84485ea712a157be] KVM: kill file->f_count abuse in kvm running grep ^SUBLEVEL = 25 Makefile SUBLEVEL = 25 Bisecting: 2740 revisions left to test after this (roughly 12 steps) [671294719628f1671faefd4882764886f8ad08cb] V4L/DVB(7879): Adding cx18 Support for mxl5005s ... ... running grep ^SUBLEVEL = 25 Makefile Bisecting: 0 revisions left to test after this (roughly 0 steps) [2ddcca36c8bcfa251724fe342c8327451988be0d] Linux 2.6.26-rc1 running grep ^SUBLEVEL = 25 Makefile 2ddcca36c8bcfa251724fe342c8327451988be0d is the first bad commit commit 2ddcca36c8bcfa251724fe342c8327451988be0d Author: Linus Torvalds <torvalds@linux-foundation.org> Date: Sat May 3 11:59:44 2008 -0700 Linux 2.6.26-rc1 :100644 100644 5cf82581... 4492984e... M Makefile bisect run success</code></pre> </div></div> <div class="paragraph"><p>In this example, we passed "grep <em>^SUBLEVEL = 25</em> Makefile" as parameter to "git bisect run". This means that at each step, the grep command we passed will be launched. And if it exits with code 0 (that means success) then git bisect will mark the current state as "good". If it exits with code 1 (or any code between 1 and 127 included, except the special code 125), then the current state will be marked as "bad".</p></div> <div class="paragraph"><p>Exit code between 128 and 255 are special to "git bisect run". They make it stop immediately the bisection process. This is useful for example if the command passed takes too long to complete, because you can kill it with a signal and it will stop the bisection process.</p></div> <div class="paragraph"><p>It can also be useful in scripts passed to "git bisect run" to "exit 255" if some very abnormal situation is detected.</p></div> </div> <div class="sect2"> <h3 id="_avoiding_untestable_commits">Avoiding untestable commits</h3> <div class="paragraph"><p>Sometimes it happens that the current state cannot be tested, for example if it does not compile because there was a bug preventing it at that time. This is what the special exit code 125 is for. It tells "git bisect run" that the current commit should be marked as untestable and that another one should be chosen and checked out.</p></div> <div class="paragraph"><p>If the bisection process is driven manually, you can use "git bisect skip" to do the same thing. (In fact the special exit code 125 makes "git bisect run" use "git bisect skip" in the background.)</p></div> <div class="paragraph"><p>Or if you want more control, you can inspect the current state using for example "git bisect visualize". It will launch gitk (or "git log" if the <code>DISPLAY</code> environment variable is not set) to help you find a better bisection point.</p></div> <div class="paragraph"><p>Either way, if you have a string of untestable commits, it might happen that the regression you are looking for has been introduced by one of these untestable commits. In this case it’s not possible to tell for sure which commit introduced the regression.</p></div> <div class="paragraph"><p>So if you used "git bisect skip" (or the run script exited with special code 125) you could get a result like this:</p></div> <div class="listingblock"> <div class="content"> <pre><code>There are only 'skip'ped commits left to test. The first bad commit could be any of: 15722f2fa328eaba97022898a305ffc8172db6b1 78e86cf3e850bd755bb71831f42e200626fbd1e0 e15b73ad3db9b48d7d1ade32f8cd23a751fe0ace 070eab2303024706f2924822bfec8b9847e4ac1b We cannot bisect more!</code></pre> </div></div> </div> <div class="sect2"> <h3 id="_saving_a_log_and_replaying_it">Saving a log and replaying it</h3> <div class="paragraph"><p>If you want to show other people your bisection process, you can get a log using for example:</p></div> <div class="listingblock"> <div class="content"> <pre><code>$ git bisect log > bisect_log.txt</code></pre> </div></div> <div class="paragraph"><p>And it is possible to replay it using:</p></div> <div class="listingblock"> <div class="content"> <pre><code>$ git bisect replay bisect_log.txt</code></pre> </div></div> </div> </div> </div> <div class="sect1"> <h2 id="_git_bisect_details">"git bisect" details</h2> <div class="sectionbody"> <div class="sect2"> <h3 id="_bisection_algorithm">Bisection algorithm</h3> <div class="paragraph"><p>As the Git commits form a directed acyclic graph (DAG), finding the best bisection commit to test at each step is not so simple. Anyway Linus found and implemented a "truly stupid" algorithm, later improved by Junio Hamano, that works quite well.</p></div> <div class="paragraph"><p>So the algorithm used by "git bisect" to find the best bisection commit when there are no skipped commits is the following:</p></div> <div class="paragraph"><p>1) keep only the commits that:</p></div> <div class="paragraph"><p>a) are ancestor of the "bad" commit (including the "bad" commit itself), b) are not ancestor of a "good" commit (excluding the "good" commits).</p></div> <div class="paragraph"><p>This means that we get rid of the uninteresting commits in the DAG.</p></div> <div class="paragraph"><p>For example if we start with a graph like this:</p></div> <div class="listingblock"> <div class="content"> <pre><code>G-Y-G-W-W-W-X-X-X-X \ / W-W-B / Y---G-W---W \ / \ Y-Y X-X-X-X -> time goes this way -></code></pre> </div></div> <div class="paragraph"><p>where B is the "bad" commit, "G" are "good" commits and W, X, and Y are other commits, we will get the following graph after this first step:</p></div> <div class="listingblock"> <div class="content"> <pre><code>W-W-W \ W-W-B / W---W</code></pre> </div></div> <div class="paragraph"><p>So only the W and B commits will be kept. Because commits X and Y will have been removed by rules a) and b) respectively, and because commits G are removed by rule b) too.</p></div> <div class="paragraph"><p>Note for Git users, that it is equivalent as keeping only the commit given by:</p></div> <div class="listingblock"> <div class="content"> <pre><code>git rev-list BAD --not GOOD1 GOOD2...</code></pre> </div></div> <div class="paragraph"><p>Also note that we don’t require the commits that are kept to be descendants of a "good" commit. So in the following example, commits W and Z will be kept:</p></div> <div class="listingblock"> <div class="content"> <pre><code>G-W-W-W-B / Z-Z</code></pre> </div></div> <div class="paragraph"><p>2) starting from the "good" ends of the graph, associate to each commit the number of ancestors it has plus one</p></div> <div class="paragraph"><p>For example with the following graph where H is the "bad" commit and A and D are some parents of some "good" commits:</p></div> <div class="listingblock"> <div class="content"> <pre><code>A-B-C \ F-G-H / D---E</code></pre> </div></div> <div class="paragraph"><p>this will give:</p></div> <div class="listingblock"> <div class="content"> <pre><code>1 2 3 A-B-C \6 7 8 F-G-H 1 2/ D---E</code></pre> </div></div> <div class="paragraph"><p>3) associate to each commit: min(X, N - X)</p></div> <div class="paragraph"><p>where X is the value associated to the commit in step 2) and N is the total number of commits in the graph.</p></div> <div class="paragraph"><p>In the above example we have N = 8, so this will give:</p></div> <div class="listingblock"> <div class="content"> <pre><code>1 2 3 A-B-C \2 1 0 F-G-H 1 2/ D---E</code></pre> </div></div> <div class="paragraph"><p>4) the best bisection point is the commit with the highest associated number</p></div> <div class="paragraph"><p>So in the above example the best bisection point is commit C.</p></div> <div class="paragraph"><p>5) note that some shortcuts are implemented to speed up the algorithm</p></div> <div class="paragraph"><p>As we know N from the beginning, we know that min(X, N - X) can’t be greater than N/2. So during steps 2) and 3), if we would associate N/2 to a commit, then we know this is the best bisection point. So in this case we can just stop processing any other commit and return the current commit.</p></div> </div> <div class="sect2"> <h3 id="_bisection_algorithm_debugging">Bisection algorithm debugging</h3> <div class="paragraph"><p>For any commit graph, you can see the number associated with each commit using "git rev-list --bisect-all".</p></div> <div class="paragraph"><p>For example, for the above graph, a command like:</p></div> <div class="listingblock"> <div class="content"> <pre><code>$ git rev-list --bisect-all BAD --not GOOD1 GOOD2</code></pre> </div></div> <div class="paragraph"><p>would output something like:</p></div> <div class="listingblock"> <div class="content"> <pre><code>e15b73ad3db9b48d7d1ade32f8cd23a751fe0ace (dist=3) 15722f2fa328eaba97022898a305ffc8172db6b1 (dist=2) 78e86cf3e850bd755bb71831f42e200626fbd1e0 (dist=2) a1939d9a142de972094af4dde9a544e577ddef0e (dist=2) 070eab2303024706f2924822bfec8b9847e4ac1b (dist=1) a3864d4f32a3bf5ed177ddef598490a08760b70d (dist=1) a41baa717dd74f1180abf55e9341bc7a0bb9d556 (dist=1) 9e622a6dad403b71c40979743bb9d5be17b16bd6 (dist=0)</code></pre> </div></div> </div> <div class="sect2"> <h3 id="_bisection_algorithm_discussed">Bisection algorithm discussed</h3> <div class="paragraph"><p>First let’s define "best bisection point". We will say that a commit X is a best bisection point or a best bisection commit if knowing its state ("good" or "bad") gives as much information as possible whether the state of the commit happens to be "good" or "bad".</p></div> <div class="paragraph"><p>This means that the best bisection commits are the commits where the following function is maximum:</p></div> <div class="listingblock"> <div class="content"> <pre><code>f(X) = min(information_if_good(X), information_if_bad(X))</code></pre> </div></div> <div class="paragraph"><p>where information_if_good(X) is the information we get if X is good and information_if_bad(X) is the information we get if X is bad.</p></div> <div class="paragraph"><p>Now we will suppose that there is only one "first bad commit". This means that all its descendants are "bad" and all the other commits are "good". And we will suppose that all commits have an equal probability of being good or bad, or of being the first bad commit, so knowing the state of c commits gives always the same amount of information wherever these c commits are on the graph and whatever c is. (So we suppose that these commits being for example on a branch or near a good or a bad commit does not give more or less information).</p></div> <div class="paragraph"><p>Let’s also suppose that we have a cleaned up graph like one after step 1) in the bisection algorithm above. This means that we can measure the information we get in terms of number of commit we can remove from the graph..</p></div> <div class="paragraph"><p>And let’s take a commit X in the graph.</p></div> <div class="paragraph"><p>If X is found to be "good", then we know that its ancestors are all "good", so we want to say that:</p></div> <div class="listingblock"> <div class="content"> <pre><code>information_if_good(X) = number_of_ancestors(X) (TRUE)</code></pre> </div></div> <div class="paragraph"><p>And this is true because at step 1) b) we remove the ancestors of the "good" commits.</p></div> <div class="paragraph"><p>If X is found to be "bad", then we know that its descendants are all "bad", so we want to say that:</p></div> <div class="listingblock"> <div class="content"> <pre><code>information_if_bad(X) = number_of_descendants(X) (WRONG)</code></pre> </div></div> <div class="paragraph"><p>But this is wrong because at step 1) a) we keep only the ancestors of the bad commit. So we get more information when a commit is marked as "bad", because we also know that the ancestors of the previous "bad" commit that are not ancestors of the new "bad" commit are not the first bad commit. We don’t know if they are good or bad, but we know that they are not the first bad commit because they are not ancestor of the new "bad" commit.</p></div> <div class="paragraph"><p>So when a commit is marked as "bad" we know we can remove all the commits in the graph except those that are ancestors of the new "bad" commit. This means that:</p></div> <div class="listingblock"> <div class="content"> <pre><code>information_if_bad(X) = N - number_of_ancestors(X) (TRUE)</code></pre> </div></div> <div class="paragraph"><p>where N is the number of commits in the (cleaned up) graph.</p></div> <div class="paragraph"><p>So in the end this means that to find the best bisection commits we should maximize the function:</p></div> <div class="listingblock"> <div class="content"> <pre><code>f(X) = min(number_of_ancestors(X), N - number_of_ancestors(X))</code></pre> </div></div> <div class="paragraph"><p>And this is nice because at step 2) we compute number_of_ancestors(X) and so at step 3) we compute f(X).</p></div> <div class="paragraph"><p>Let’s take the following graph as an example:</p></div> <div class="listingblock"> <div class="content"> <pre><code> G-H-I-J / \ A-B-C-D-E-F O \ / K-L-M-N</code></pre> </div></div> <div class="paragraph"><p>If we compute the following non optimal function on it:</p></div> <div class="listingblock"> <div class="content"> <pre><code>g(X) = min(number_of_ancestors(X), number_of_descendants(X))</code></pre> </div></div> <div class="paragraph"><p>we get:</p></div> <div class="listingblock"> <div class="content"> <pre><code> 4 3 2 1 G-H-I-J 1 2 3 4 5 6/ \0 A-B-C-D-E-F O \ / K-L-M-N 4 3 2 1</code></pre> </div></div> <div class="paragraph"><p>but with the algorithm used by git bisect we get:</p></div> <div class="listingblock"> <div class="content"> <pre><code> 7 7 6 5 G-H-I-J 1 2 3 4 5 6/ \0 A-B-C-D-E-F O \ / K-L-M-N 7 7 6 5</code></pre> </div></div> <div class="paragraph"><p>So we chose G, H, K or L as the best bisection point, which is better than F. Because if for example L is bad, then we will know not only that L, M and N are bad but also that G, H, I and J are not the first bad commit (since we suppose that there is only one first bad commit and it must be an ancestor of L).</p></div> <div class="paragraph"><p>So the current algorithm seems to be the best possible given what we initially supposed.</p></div> </div> <div class="sect2"> <h3 id="_skip_algorithm">Skip algorithm</h3> <div class="paragraph"><p>When some commits have been skipped (using "git bisect skip"), then the bisection algorithm is the same for step 1) to 3). But then we use roughly the following steps:</p></div> <div class="paragraph"><p>6) sort the commit by decreasing associated value</p></div> <div class="paragraph"><p>7) if the first commit has not been skipped, we can return it and stop here</p></div> <div class="paragraph"><p>8) otherwise filter out all the skipped commits in the sorted list</p></div> <div class="paragraph"><p>9) use a pseudo random number generator (PRNG) to generate a random number between 0 and 1</p></div> <div class="paragraph"><p>10) multiply this random number with its square root to bias it toward 0</p></div> <div class="paragraph"><p>11) multiply the result by the number of commits in the filtered list to get an index into this list</p></div> <div class="paragraph"><p>12) return the commit at the computed index</p></div> </div> <div class="sect2"> <h3 id="_skip_algorithm_discussed">Skip algorithm discussed</h3> <div class="paragraph"><p>After step 7) (in the skip algorithm), we could check if the second commit has been skipped and return it if it is not the case. And in fact that was the algorithm we used from when "git bisect skip" was developed in Git version 1.5.4 (released on February 1st 2008) until Git version 1.6.4 (released July 29th 2009).</p></div> <div class="paragraph"><p>But Ingo Molnar and H. Peter Anvin (another well known linux kernel developer) both complained that sometimes the best bisection points all happened to be in an area where all the commits are untestable. And in this case the user was asked to test many untestable commits, which could be very inefficient.</p></div> <div class="paragraph"><p>Indeed untestable commits are often untestable because a breakage was introduced at one time, and that breakage was fixed only after many other commits were introduced.</p></div> <div class="paragraph"><p>This breakage is of course most of the time unrelated to the breakage we are trying to locate in the commit graph. But it prevents us to know if the interesting "bad behavior" is present or not.</p></div> <div class="paragraph"><p>So it is a fact that commits near an untestable commit have a high probability of being untestable themselves. And the best bisection commits are often found together too (due to the bisection algorithm).</p></div> <div class="paragraph"><p>This is why it is a bad idea to just chose the next best unskipped bisection commit when the first one has been skipped.</p></div> <div class="paragraph"><p>We found that most commits on the graph may give quite a lot of information when they are tested. And the commits that will not on average give a lot of information are the one near the good and bad commits.</p></div> <div class="paragraph"><p>So using a PRNG with a bias to favor commits away from the good and bad commits looked like a good choice.</p></div> <div class="paragraph"><p>One obvious improvement to this algorithm would be to look for a commit that has an associated value near the one of the best bisection commit, and that is on another branch, before using the PRNG. Because if such a commit exists, then it is not very likely to be untestable too, so it will probably give more information than a nearly randomly chosen one.</p></div> </div> <div class="sect2"> <h3 id="_checking_merge_bases">Checking merge bases</h3> <div class="paragraph"><p>There is another tweak in the bisection algorithm that has not been described in the "bisection algorithm" above.</p></div> <div class="paragraph"><p>We supposed in the previous examples that the "good" commits were anc