(Collaborative post by Mateusz “j00ru” Jurczyk and Gynvael Coldwind)

Several months ago, we started an internal Google Security Team effort to improve the general security posture of the Chrome embedded PDF reader, in an approach similar to the Flash fuzzing performed several months ago by Tavis Ormandy. During the course of a few weeks, we built a solid corpus of PDF documents that we feel gets significant coverage of the Chrome PDF Reader’s code base and used it to shake out more than 50 low-to-high severity bugs. All of the high and critical severity bugs we discovered have been fixed in the stable channel [1] [2] [3] as of this posting; see examples:

[132585] [132694] [132861] High CVE-2012-2851: Integer overflows in PDF viewer. Credit to Mateusz Jurczyk of Google Security Team, with contributions by Gynvael Coldwind of Google Security Team.

[134888] High CVE-2012-2855: Use-after-free in PDF viewer. Credit to Mateusz Jurczyk of Google Security Team, with contributions by Gynvael Coldwind of Google Security Team.

[134954] [135264] High CVE-2012-2856: Out-of-bounds writes in PDF viewer. Credit to Mateusz Jurczyk of Google Security Team, with contributions by Gynvael Coldwind of Google Security Team.

[136643] [137721] [137957] High CVE-2012-2862: Use-after-free in PDF viewer. Credit to Mateusz Jurczyk of Google Security Team, with contributions by Gynvael Coldwind of Google Security Team.

[136968] [137361] High CVE-2012-2863: Out-of-bounds writes in PDF viewer. Credit to Mateusz Jurczyk of Google Security Team, with contributions by Gynvael Coldwind of Google Security Team.

Given the success of our corpus against Chrome’s PDF Reader, we decided to use it to test another widely-used PDF viewing application - Adobe Reader 9.5.1 for GNU/Linux (latest version available for the platform). Over 1,500 cores have been continuously feeding the application with malformed data for a few weeks, which ultimately resulted in a total of 46 reproducible crashes with unique stack traces. This initial batch of files was sent directly to Adobe on 21st of June.  A few days later, we came up with a slightly different mutation method with the potential of exposing additional software vulnerabilities and re-ran the fuzzer. As a direct outcome, 14 new unique crashes were identified and sent to Adobe for further evaluation on the 27th of June. In addition to sending information to the vendor, we also performed a cursory investigation of the crash logs and determined that 31 (roughly 50%) seemed to represent trivially exploitable problems, which we assess as critical bugs, and 9 test-cases as potentially exploitable for remote code execution.

Since our original disclosure, we have been in regular contact with the Adobe PSIRT team. They were immediately responsive to our report and have provided us with regular updates about their progress addressing these bugs and update plans. We appreciate the progress they’ve made on the multiple crashes reported.

Today, on 14th of August 2012, Adobe has released a new version of Reader for Windows and Mac OS X platforms, addressing around 25 of the reported critical crashes, see the APSB12-16 security bulletin. The issues were assigned twelve CVE’s in total (CVE-2012-4149 through CVE-2012-4160), indicating how many unique code changes it took to fix the problems. Fixing those and numerous other lower-severity bugs not mentioned here in less than two months is a great step forward in raising the bar for bug hunters and improving users’ safety worldwide.

Unfortunately, sixteen more crashes affecting Windows, OS X, or both systems remain unpatched. Considering that fixing the first twenty four crashes took twelve unique code fixes, it is expected that the remaining crashes might represent around eight more unique problems. Adobe plans to fix these remaining bugs and issue an update for the Linux version of Reader in an upcoming release. Though we have no evidence these bugs are being exploited today, we are concerned that functional exploits can be built without much effort based on knowledge derived from binary diffing of the old and newly patched Windows builds.

Given this, we consider users of Adobe Reader to be exposed to serious risk. Using our thoughts on reasonable disclosure as a guide, we notified Adobe of our plans to publicly disclose information about any critical vulnerabilities which would remain unfixed 60 days beyond our initial contact. (Note: Adobe has confirmed they have no plans to issue additional out of band updates before August 27, which is 60 days after we disclosed all bugs. Since the Linux Reader version remains unpatched and the Windows / OS X patches are now available for diffing and reverse engineering, we have decided that it’s in the best interest of users to be aware of these security issues without additional delay.)

It is important to note that all discussed vulnerabilities were found using publicly available PDF documents, altered using conceptually trivial mutation algorithms such as bitflipping. Given that, we believe it is very possible that third-parties specializing in bug hunting and vulnerability research may already know of and/or be targeting many of our reported issues.

We plan to continue working with Adobe to verify additional fixes and test new releases to further improve the security of Reader.

To summarize:
  • Adobe Reader for Linux users are exposed to all critical vulnerabilities discussed here, until the patched Linux version is released.
  • Adobe Reader for Windows are currently vulnerable to up to 6 unpatched issues.
  • Adobe Reader for Mac OS X are currently vulnerable to up to 10 unpatched issues.

Vulnerability information

We have decided to publish the stack traces of all sixteen crashes affecting Windows and OS X, with the intention of demonstrating the existence and severity of the issues. The call stacks are, however, obfuscated in such a way that the 20 least-significant address bits are masked out together with function symbols and any other meaningful information that might be used by third parties to directly locate the vulnerable code path.

ar_callstack.txt

Workarounds and mitigation

Two of the discussed vulnerabilities affecting Reader for Linux have been confirmed to reside in the Annots.api and PPKLite.api plugins, respectively. Since no documented ways of disabling specific plugins are available, users are advised to remove these two files from their /path/to/Adobe/Reader9/Reader/intellinux/plug_ins directory as a workaround for the issues.

There are currently no known workarounds available against any of the remaining unpatched vulnerabilities. If you believe you may be affected, you may wish to do one of the following until the patches have been released:
  • Limit the use of Adobe Reader software.
  • Or at least, do not open any externally received PDF documents.
  • Disable the Adobe Reader browser extension for the time being.
Users of Adobe Reader 9.x for Windows who are aware of the risk are advised to upgrade to Adobe Reader X, which provides a sandbox feature, making it more difficult (although not impossible) to exploit these vulnerabilities. Unfortunately, the sandbox feature is not available for the newest versions of Adobe Reader for OS X or Linux.

Timeline
  • June 2012 - discovery of the first set of crashes
  • 21st of June 2012 - first set of crashes reported to vendor
  • 26th of June 2012 - we notify Adobe that all critical crashes would be subject to the 60-day policy
  • 27th of June 2012 - second set of crashes reported to vendor
  • July 2012 - e-mails back and forth, we are notified not every critical issue would be fixed
  • 14th of August 2012 - new version for Windows and OS X released, we publish this post

Comments:

2012-08-14 23:52:53 = carstein
{
+1
}
2012-08-15 09:27:13 = svenne
{
can you make the base fileset available? I have a collection of "publically available pdf files" myself, and i'd like to see how much overlap there is. Further; did you do any tracing for code coverage on the base sets, and if so, how?
}
2012-08-15 11:50:50 = Nism0
{
Great work guys ! Keep the fire burning :] Congratulations ;>
}
2012-08-15 12:19:06 = j00ru
{
@svenne: The secret in finding so many vulnerabilities is the file corpus itself and the coverage it provides - no advanced or intelligent mutation techniques were applied. Therefore, we are not going to make it publicly available at least until we are definitely sure that it can't be used to easily find critical bugs in commonly used software - and as mentioned in the post, even now there are a few of them known to us but unpatched by Adobe.

To answer your other question, the corpus we used was created specifically by performing code coverage analysis over a large number of PDF documents, and later distillating it to obtain the smallest set of files with the best coverage we could achieve. Dynamic binary instrumentation was employed for that purpose; we might share some more technical details of the overall process in the future.

Cheers!
}
2012-08-15 12:46:20 = Grzechooo
{
And that's why I don't use Adobe Reader.
}
2012-08-15 14:34:42 = svenne
{
I understand why you won't make the fileset available now. I am interested in your method for DBI; I encounter strange problems using PIN / minset from peach and a lot of other tried methods, for example tracing the same file 10 times results in a minset of ~8 files :/
}
2012-08-15 15:12:21 = hello
{
PDF Xchange is a stable comprehensive PDF reader, but it's probably not been tested yet. Would you consider testing it?

115 million users they claim.
}
2012-08-15 20:03:30 = jviereck
{
Have you tested the PDF.JS PDF viewer with your test PDFs?

See:
* online viewer: http://mozilla.github.com/pdf.js/web/viewer.html
* "homepage": https://github.com/mozilla/pdf.js
}
2012-08-16 00:13:38 = Zirgin
{
In your opinien what's the best alternative for Adobe Reader (exclude Chrome PDF Viewer ^^)?
}
2012-08-16 08:20:52 = Bartek
{
Great work!
Java, QuickTime, Shockwave, Silverlight, Foxit anyone?
}
2012-08-16 09:15:53 = TB
{
Would it be possible to test the PDF files also against poppler, which seems to be the most popular open-source PDF engine under Linux, used e.g. by KDE's okular and some Gnome viewers.
}
2012-08-16 11:37:49 = j00ru
{
@svenne: Interesting. I can acknowledge that making use of the PIN/DynamoRIO frameworks should be relatively easy and definitely suffices for any small-to-medium code coverage analysis project (although the overhead for both is quite significant). I think it is possible to build an instrumentation tool based just on the examples and available documentation. Good luck ;)

@hello: I find it hard to believe that number. We might consider testing it, time will show.

@jviereck: Not yet - the impact of crashing a random Javascript PDF parser seems to be a lot smaller than native Google Chrome / Adobe Reader. Thanks for your suggestion though!

@Zirgin: Most (if not all) currently available PDF readers appear to have security issues. It's really hard to recommend something, but I'd really insist on Chrome ;)

@Bartek: Stay tuned...

@TB: See http://j00ru.vexillium.org/?p=1175#comments. I suppose we have to resume our attempts to have the project fixed.
}
2012-08-16 11:38:49 = Gynvael Coldwind
{
@hello
Can't promise anything, though if there is a Linux version we might sneak a peek :)

@jviereck
Since it's, as the name suggest, a javascript/HTML5 implementation of a PDF reader and renderer, there is little sense in testing it using fuzzing of the kind we use.
Unless, you want to indirectly look for bugs in the underlying JS engine that is :)
I guess more sense would make looking for XSSes in this case :)

@Zirgin
Actually I don't have a good answer for that question. Personally I use Chrome's internal PDF viewer, though I don't read any fancy 3D-graphics/forms-with-signing/etc kind of features, so it's good enough for me.
I wouldn't recommend Xpdf since, if I recall correctly, there are a lot of easily discoverable bugs and last version was over a year ago, so it doesn't seem to be actively developed.
Also, see the note about poppler below.

@Bartek
;)

@TB
We did test poppler and found heaps of bugs. We've sent them to poppler maintainers some time ago but they have only fixed a few so far:
http://cgit.freedesktop.org/poppler/poppler/log/?qt=grep&q=Mateusz+%22j00ru%22+Jurczyk+and+Gynvael+Coldwind
}
2012-08-16 13:41:50 = sandsmark
{
thanks for testing poppler!

nice to know that you care about the rest of the open source eco-system as well. :)
}
2012-08-16 14:13:07 = QrA
{
świetna robota. jak zwykle ;) czekam na kolejne posty. pozdrawiam!
}
2012-08-16 14:49:13 = jviereck
{
@j00ru

> @jviereck: Not yet - the impact of crashing a random Javascript PDF parser seems to be a lot smaller than native Google Chrome / Adobe Reader. Thanks for your suggestion though!

PDF.JS is about to become the default PDF viewer in Firefox - it's already the default viewer in the beta/aurora/nightly channel. Firefox has around 450-500M daily users and it would be good to know if they are at potential risk or not.

Although the viewer is written in JS/HTML/CSS, it might still be possible to exploit it. The fonts and images might have corrupted data, that cause the browser to crash. All browsers test against corrupted data, but maybe not good enough.
}
2012-08-17 10:50:30 = Infern0_
{
Standardowo wielkie gratulacje, tutaj art o Was:

http://thehackernews.com/2012/08/google-engineers-warn-of-serious.html

Pozdro.
}
2012-08-17 21:14:02 = Anonymous
{
What about mupdf? Can we get this corpus to test other pdf implementations? It'd be great if such effective work could be shared freely and help the community at large.
}
2012-08-17 22:56:53 = davidwr
{
While I applaud your efforts, it would've been "cooler" and "less evil" to honor your original commitment to Adobe and limit yourself to a general announcement now with a promise to release the details you released today on August 27.

Such a public statement could be enough to get Adobe to change its mind and do an out-of-band patch.

Remember, if you have "don't be evil" as an unofficial corporate motto, people expect you to honor your commitments.
}
2012-08-18 00:21:41 = Gynvael Coldwind
{
@QrA
Thx ;)

@jviereck
I think you're actually suggesting to fuzz the browser, and not the PDF.JS app itself :)

@Infern0_
Thx ;)

@Anonymous
As said before, if there is a linux version of that client, there is a possibility that we'll take a look at it.

@davidwr
Your comment is based on the belief that Adobe would issue an out-of-band patch if we release "a general announcement" - I'm afraid that belief is incorrect, though I would like it to be true. Unfortunately Adobe was not interested in releasing an out-of-band update to fix the remaining issues and it seems the next normal update will not take place any time soon.

Please also take a look on our thoughts about the disclosure - http://googleonlinesecurity.blogspot.ch/2010/07/rebooting-responsible-disclosure-focus.html
}
2012-08-20 13:11:35 = DJViking
{
What of the Adobe Reader for Android? I also use ezPDF Reader on Android.
}
2012-08-21 18:29:56 = RJJ
{
Ghostscript is still around also, and is used for printing PDF's on many linux systems. We (Artifex Software) would welcome any bug reports of security exploits. To send private email, you can send to support @ artifex.com (even though that is usually reserved for licensed customer bug reports).
}
2012-08-29 23:02:21 = JRD
{
So, how can we catch malformed PDFs on the wire? There doesn't seem to be a signature or regex that can be written to account for all the possibilities. Since you just fuzzed around you could probably fuzz around any regex.
Is the only recourse to update to X or remove and replace it?
}
2012-10-18 08:29:34 = saso
{
Would be great if you would consider to do some fuzzing like this on LibreOffice...
}

Add a comment:

Nick:
URL (optional):
Math captcha: 10 ∗ 8 + 8 =