Archive for the ‘Python’ Category

ropemacs and remote files (fuse)

November 16, 2010

This morning I needed to edit some python scripts from a local server at work and, as always, used the TRAMP Emacs mode, but immediately found a problem. A few months ago I installed the great python refactoring library rope and its Emacs mode ropemacs. It comes out that ropemacs asks you for the location of your rope project if it cannot find it (this always happens the first time you start a project).

TRAMP URLs look like this:

/ssh:user@server:/path

So, you can now imagine what happens when Emacs gives an URL like this to rope (which is a python library and doesn’t know anything about TRAMP URLs) as if it was a local file name…

Traceback (most recent call last):
  ...
  ...
  File "/usr/lib/python2.6/.../ropemode/interface.py", line 88, in open_project
    self.project = rope.base.project.Project(root)
  File "/usr/lib/python2.6/.../rope/base/project.py", line 134, in __init__
    os.mkdir(self._address)
OSError: [Errno 2] No such file or directory: '/ssh:user@server:/path'

The worst is that you are continuously being asked by Emacs to enter a valid rope project location.

The solution? Simply use the FUSE SSH Filesystem.

$ sudo apt-get install sshfs
$ sshfs user@server:path mountpoint

FUSE stands for Filesystem in Userspace. With FUSE you will end up editing remote files as if they were in your computer and the great thing is that you don’t need root access, so any user can mount a FUSE filesystem.

Update 2010/11/17: FUSE mounted directory can be unmounted with:

$ fusermount -u mountpoint
Advertisements

More on easy acronyms generation

October 22, 2010

About four years ago I posted an article about easy acronyms generation in LaTeX. Yesterday, I did some updates to the script (tex-acronyms.py) that I wanted to share with you (if anyone is reading…). Basically, the updates are:

  • A user defined acronyms file can be specified via the -u argument. User defined acronyms take precedence over global acronyms definition.
  • Global excluded acronyms file has been removed. Now, the user must define acronyms to be excluded in the user defined acronyms file as an empty acronym. For example:

    \nomenclature{GHH}{}
    

    Then, the definition of GHH will not be included in the list of acronyms.

  • To facilitate the read of acronyms conflicts, it is now specified if the conflict is because of a “Duplicated“, “Undefined” or “Excluded” acronym.

So, the script is called as before but a new optional argument -u can be specified for the user defined acronyms file:

tex-acronyms.py [-r] -d /path/to/acronyms \
                -i article.tex \
                [-u user_acronyms.tex] \
                -o acronyms.tex \
                -e acronyms.errors

BitPacket: Python 2.x and 3.0 compatibility

October 14, 2010

Lately, I’ve been porting BitPacket to Python 3.0. I wanted to keep backwards compatibility with Python 2.6 (which is the 2.x I have in my Debian) and, thankfully, I only had to fix three minor issues:

  • Unicode strings
  • Dictionary keys
  • Bytes vs. strings

StringIO and unicode strings

If you have ever used the StringIO module you should be familiar with this:

try:
    from cStringIO import StringIO
except ImportError:
    from StringIO import StringIO

In Py3k the StringIO is located under the io package, so you should changed the above by:

from io import StringIO

which is also compatible with Python 2.6.

Once I did the change my code only worked in Py3k, Python 2.6 complained when trying to use the write method with a simple string:

>>> from io import StringIO
>>> stream = StringIO()
>>> stream.write("test")
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python2.6/io.py", line 1515, in write
    s.__class__.__name__)
TypeError: can't write str to text stream

You should note that in Py3k all the strings are unicode strings by default. This is not true in Python 2.6, so my first approach was the following:

>>> stream.write(u"test")
4

Unfortunately, this only worked in Python 2.6. Py3k does not recognize the unicode prefix “u“, giving you this error:

>>> stream.write(u"test")
  File "", line 1
    stream.write(u"test")
                       ^
SyntaxError: invalid syntax

I googled a bit and find out a Making code compatible with Python 2 and 3 post (from the guy that finished all the SICP exercices) where it explained some similar issues, so I came up with this solution:

def u_str(string):
    if sys.hexversion >= 0x03000000:
        return string
    else:
        return unicode(string)

>>> stream.write(u_str("test"))
4

In Py3k, unicode does not exist but as that line is never executed we don’t get any error.

Even that worked well, I was not very happy with it. It was too slow and I had to use the custom u_str function everywhere. So, I googled a bit more and I found a nice pycon 2009 talk about Python 3.0 compatibility. Finally, I had which I think is the best solution (for both speed and clearness):

try:
    # This will raise an exception in Py3k, as unicode doesn't exist
    str = unicode
except:
    pass

So, instead of defining a new u_str function, the str type is re-defined as unicode for Python 2.6. Then, I only had to update all the strings in the code to use str:

>>> stream.write(str("test"))
4

Note: I put this code in a compatibility.py file and import it everywhere I need it.

Dictionary keys

The next problem was reported by the 2to3 tool that comes with Py3k.

-                for k in field.keys():
+                for k in list(field.keys()):

Basically, it told me that the dictionary keys() method returns a view in Py3k not a list, so it needs to be converted to a list as explained here:

dict methods dict.keys(), dict.items() and dict.values() return “views” instead of lists. For example, this no longer works: k = d.keys(); k.sort(). Use k = sorted(d) instead (this works in Python 2.5 too and is just as efficient).

Bytes vs. strings

Finally, the last issue was about the difference between strings and bytes in Python 2.x and 3.0. In Python 2.x, bytes is just an alias for str:

>>> bytes
<type 'str'>

In Py3k, bytes and str are different classes and behave differently, see below:

>>> s = "AB"
>>> s[0]
'A'
>>> s[1]
'B'
>>> b = b"AB"
>>> b[0]
65
>>> b[1]
66

This means that one needs to take care of functions returning bytes (e.g. struct.pack) and the operations performed with the returned data, in my case a call to the ord function, that failed with the typical error message:

TypeError: ord() expected string of length 1, but int found

So, following the approaches mentioned above I added the following function to my compatibility.py:

def u_ord(c):
    if sys.hexversion >= 0x03000000:
        return c
    else:
        return ord(c)

which I used instead of the built-in ord in the struct.pack case.

Hope this helps to someone.

Happy hacking!

Handling external dependencies with Mercurial

March 17, 2008

In order to add external repository dependencies in Mercurial, I have created an extension similar to the multi-project management feature of GNU arch. This can be probably done with other existent extensions, but I have not been able to found one that exactly met my needs (the Forest extension can be used for non-versioned Mercurial dependencies, and, in fact, they could be used together, but it is not exactly the same). So please, if anyone knows similar extensions, I would happy to know it. My friend LluĂ­s gave me the idea to also handle non-Mercurial dependencies, so thanks from here!

If interested, check the Deps extension.

Update 2008/04/28: I have removed the extension help from this entry and added a Deps extension page to Mercurial’s wiki.

From GNU Arch to Mercurial

January 31, 2008

gnuarch.pngOver the last two and a half years we (at work) have been using baz as our distributed version control system. Baz is based on tla, the original GNU arch implementation by Tom Lord. After all this time we have applied more than two thousand patches (distributed in various projects). xtla.pngWe also have been using Xtla (a wonderful Emacs front-end to GNU Arch) and lately DVC (its successor, which handles GNU Arch, Bazaar, git, Mercurial and Monotone). After some months thinking about it, I decided that it was time for a change, baz was too slow working with this magnitude of patches and even slower in my PowerBook. I also wanted a fast web interface and ArchZoom was not an option as it took ages to show changes between two revisions and mostly in any operation. So, I watched the Mercurial Google TechTalk and I also read some blog entries about git and Bazaar. dvc.png I decided to give Mercurial a try as it also had complete DVC support and a great web interface (like the darcs one).

One of the things I needed was a tool to convert our baz repositories to Mercurial. As far as I knew, there was no such a tool, so I thought I could use arch2darcs to convert the repositories to darcs and then use the Mercurial convert extension to convert them to Mercurial. I tried it but it didn’t work. The Mercurial convert extension worked really well with darcs for my personal projects, so, why not write a GNU Arch converter?

After some days trying it, I have posted an issue to Mercurial BTS with a GNU Arch converter. It currently does not support directory renaming (fixed, see updates), but all other things I have tried work well. mercurial.png The converter is slow as I didn’t know how to compute a complete changeset between patches easily (any help?… solved, see updates). The log files might not provide all the information needed (added, removed, modified files), so I decided to use the delta command which computes all this information, and this takes a bit of time.

To use the converter (in 0.9.5) copy it to the convert extension directory (hgext/convert) and edit the __init__.py file following, for example, the darcs converter (you’ll easily see how to add it). Use the converter at your own risk, as I am sure it needs more work. I would appreciate any comments and bug reports until it is included in Mercurial (if ever).

Update 2008/04/28: The GNU Arch converter is available since Mercurial 1.0.

Update 2008/02/09: The converter does not use the delta command anymore to compute a changeset (the replay command used to merge already does it). This has increased speed drastically (in the order of hours).

Update 2008/02/06: The GNU Arch converter has been added to Mercurial development branch. Thanks!

Update 2008/02/04: Directory renaming is now supported. A bug related to link modifications has been fixed. The ‘–rev’ option is also supported. You will need to download and install the development branch (hg clone http://hg.intevation.org/mercurial/crew) and apply the patch provided here.

Update 2008/02/01: Yesterday, I found this page (weird, as I didn’t see it some days ago) which talks about GNU Arch conversions to Mercurial. None of the converters uses the convert extension, but I guess it would had been easier to reuse them.