title: ReviewBoard: symbol XML_SetHashSalt, version EXPAT_2_0_1_RH not defined in file libexpat.so.1 with link time reference
body: <p>I'm running ReviewBoard 2.0.15 and on some reviews I'm getting the following error:</p>
  
  <pre><code>Django Version: 1.6.11
  Python Version: 2.6.6
  Installed Applications:
  [u'django.contrib.admin',
   u'django.contrib.auth',
   u'django.contrib.contenttypes',
   u'django.contrib.sites',
   u'django.contrib.sessions',
   u'django.contrib.staticfiles',
   u'djblets',
   u'djblets.configforms',
   u'djblets.datagrid',
   u'djblets.extensions',
   u'djblets.feedview',
   u'djblets.gravatars',
   u'djblets.log',
   u'djblets.pipeline',
   u'djblets.siteconfig',
   u'djblets.util',
   u'djblets.webapi',
   u'haystack',
   u'pipeline',
   u'reviewboard',
   u'reviewboard.accounts',
   u'reviewboard.admin',
   u'reviewboard.attachments',
   u'reviewboard.changedescs',
   u'reviewboard.datagrids',
   u'reviewboard.diffviewer',
   u'reviewboard.extensions',
   u'reviewboard.hostingsvcs',
   u'reviewboard.notifications',
   u'reviewboard.reviews',
   u'reviewboard.reviews.ui',
   u'reviewboard.scmtools',
   u'reviewboard.site',
   u'reviewboard.ssh',
   u'reviewboard.webapi',
   u'django_evolution']
  Installed Middleware:
  [u'django.middleware.gzip.GZipMiddleware',
   u'reviewboard.admin.middleware.InitReviewBoardMiddleware',
   u'django.middleware.clickjacking.XFrameOptionsMiddleware',
   u'django.middleware.common.CommonMiddleware',
   u'django.middleware.doc.XViewMiddleware',
   u'django.middleware.http.ConditionalGetMiddleware',
   u'django.middleware.locale.LocaleMiddleware',
   u'django.contrib.sessions.middleware.SessionMiddleware',
   u'django.contrib.auth.middleware.AuthenticationMiddleware',
   u'django.contrib.messages.middleware.MessageMiddleware',
   u'djblets.siteconfig.middleware.SettingsMiddleware',
   u'reviewboard.admin.middleware.LoadSettingsMiddleware',
   u'djblets.extensions.middleware.ExtensionsMiddleware',
   u'djblets.log.middleware.LoggingMiddleware',
   u'reviewboard.accounts.middleware.TimezoneMiddleware',
   u'reviewboard.admin.middleware.CheckUpdatesRequiredMiddleware',
   u'reviewboard.admin.middleware.X509AuthMiddleware',
   u'reviewboard.site.middleware.LocalSiteMiddleware',
   u'djblets.extensions.middleware.ExtensionsMiddlewareRunner',
   u'reviewboard.admin.middleware.ExtraExceptionInfoMiddleware']
  
  
  Traceback:
  File "/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/Django-1.6.11-py2.6.egg/django/core/handlers/base.py" in get_response
    112.                     response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/ReviewBoard-2.0.15-py2.6.egg/reviewboard/accounts/decorators.py" in _check
    23.             return view_func(*args, **kwargs)
  File "/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/ReviewBoard-2.0.15-py2.6.egg/reviewboard/site/decorators.py" in _check
    35.         return view_func(request, local_site=local_site, *args, **kwargs)
  File "/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/ReviewBoard-2.0.15-py2.6.egg/reviewboard/reviews/views.py" in review_detail
    665.                         changedesc.fields_changed[field_id])
  File "/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/ReviewBoard-2.0.15-py2.6.egg/reviewboard/reviews/fields.py" in get_change_entry_sections_html
    200.             'rendered_html': mark_safe(self.render_change_entry_html(info)),
  File "/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/ReviewBoard-2.0.15-py2.6.egg/reviewboard/reviews/fields.py" in render_change_entry_html
    590.         old_lines = list(iter_markdown_lines(old_value))
  File "/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/ReviewBoard-2.0.15-py2.6.egg/reviewboard/reviews/markdown_utils.py" in iter_markdown_lines
    212.     nodes = get_markdown_element_tree(markdown_html)
  File "/opt/local/revboard/python/virtualenv/lib/python2.6/site-packages/ReviewBoard-2.0.15-py2.6.egg/reviewboard/reviews/markdown_utils.py" in get_markdown_element_tree
    274.     doc = parseString(b'&lt;html&gt;%s&lt;/html&gt;' % markdown_html)
  File "/usr/lib64/python2.6/xml/dom/minidom.py" in parseString
    1927.         from xml.dom import expatbuilder
  File "/usr/lib64/python2.6/xml/dom/expatbuilder.py" in &lt;module&gt;
    32. from xml.parsers import expat
  File "/usr/lib64/python2.6/xml/parsers/expat.py" in &lt;module&gt;
    4. from pyexpat import *
  
  Exception Type: ImportError at /reviews/r/24291/
  Exception Value: /usr/lib64/python2.6/lib-dynload/pyexpat.so: symbol XML_SetHashSalt, version EXPAT_2_0_1_RH not defined in file libexpat.so.1 with link time reference
  </code></pre>
  
  <p>If I run python using the same virtual environment manually, it works fine:</p>
  
  <pre><code>bin]$ python
  Python 2.6.6 (r266:84292, Jan 22 2014, 01:49:05) 
  [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  &gt;&gt;&gt; from pyexpat import *
  &gt;&gt;&gt; 
  </code></pre>
  
  <p>I have verified that apache and python do use the exact same version of expat, so that can't be the problem either. I'm really confused as to why this would happen on only a handful of reviews out of several 100.</p>
  
  <p>EDIT: I ran some more tests and I noticed some stuff.</p>
  
  <p>I did notice one thing. The libexpat.so that apache is using points to /usr/local/lib/libexpat.so, and if I run the following on that file I get no results:</p>
  
  <pre><code>(virtualenv)[user@rbdev301 lib]$ strings libexpat.so.1.5.2 | grep XML_SetHashSalt
  (virtualenv)[user@rbdev301 lib]$ strings libexpat.so.1.5.2 | grep EXPAT_2_0_1_RH
  </code></pre>
  
  <p>However, when I check which expat Python is using, I get this:</p>
  
  <pre><code>(virtualenv)[user@rbdev301 lib-dynload]$ ldd pyexpat.so 
  linux-vdso.so.1 =&gt;  (0x00007ffff252a000)
  libexpat.so.1 =&gt; /lib64/libexpat.so.1 (0x00007f52afbae000)
  libpython2.6.so.1.0 =&gt; /usr/lib64/libpython2.6.so.1.0 (0x00007f52af802000)
  libpthread.so.0 =&gt; /lib64/libpthread.so.0 (0x00007f52af5e4000)
  libc.so.6 =&gt; /lib64/libc.so.6 (0x00007f52af250000)
  libdl.so.2 =&gt; /lib64/libdl.so.2 (0x00007f52af04c000)
  libutil.so.1 =&gt; /lib64/libutil.so.1 (0x00007f52aee48000)
  libm.so.6 =&gt; /lib64/libm.so.6 (0x00007f52aebc4000)
  /lib64/ld-linux-x86-64.so.2 (0x00007f52affe3000)
  </code></pre>
  
  <p>So, python is using the one in /lib64. If I run the same commands on that one, I do get results.</p>
  
  <pre><code>(virtualenv)[user@rbdev301 lib64]$ strings libexpat.so.1.5.2 | grep XML_SetHashSalt
  XML_SetHashSalt
  (virtualenv)[user@rbdev301 lib64]$ strings libexpat.so.1.5.2 | grep EXPAT_2_0_1_RH
  EXPAT_2_0_1_RH
  </code></pre>
  
  <p>The one that Apache is using was compiled from the same source as the one that python is using, and they're both the same version. It just seems that Python is using the 64bit one and Apache is not? Maybe I'm on a wild goose chase here.</p>