jump to content

Convert files and images to extfile/image

Jump to the code.

Introduction

Most OS'es impose a limit of 2GB for a Zope based site because Zope's object database (ZODB) stores everything in a single file - and in most OS, a file can't be larger than 2GB. This is not ideal for intranet solutions where people have an unstoppable desire to upload large binary documents - MS Office files, Zip files, reports, downloadable software etc. Before you come up with this brilliant idea - no, you can't educate your users that nobody is going to download a 8MB Excel file. I tried with log files and comments from irrate intranet visitors, but the content managers just want to feel good about posting huge documents :-)

How ExtFile/Image can address my problem?

ExtFile/Image is a great Zope product whereby you can store the actual binary data on the file system and still have it accessible as Zope objects with properties you can set.

For more information on Extfile/Image, please see the product home pageexternal link . You may also see Jon Udellexternal link 's article on mixed storageexternal link .

With hard disk prices so cheap these days, we can have pretty big drives that can store quite a lot of data. So, what I figured out was to implement ExtFile/Image instead of normal Zope objects for File/Image. We already had a lot of files and images stored in ZODB, uploaded by users using custom forms - not Zope's default "Add Object" method. We could easily rework those forms to set the future uploads to be of ExtFile/Image types.

Next problem - How to convert existing files/images?

I banged my head against the wall for some time, posted a message to Zope mailing list which nobody took interest in. Then I wrote to the author of ExtFile/Image product and he sent me a code for a folderish product ExtFileFolder that overwrites PUT_Factory method so that all future PUT/FTP uploads will create ExtFile/Images instead of File/Image. That is a cool concept. Only two problems - PUT_Factory works well for almost anything else, but not for these two :-( Moreover, we don't provide PUT/FTP access to our Zope. Then I thought of writing a Python program that runs once, downloading all these files and then call IE's COM automation to upload these using our new forms - *really inelegant*.

Finally, I thought of writing an external method that will do this. I couldn't understand why a simple manage_addProduct['ExtFile'].manage_addExtFile() wouldn't work. Some weird errors about stream type, things failing in _copy() function within the product. Thanks to Python's good error messages, I figured out that it was failing in finding a stream.

Solution

I wrote an external method zodb2ext.py that is passed a Zope object. It deletes the Zope object, and recreates the corresponding ExtFile/Image object.

A small python script convert2ext loops through all the objects in a folder and call this external method for each File and Image.

External Method - zodb2ext.py
"""zodb2ext

Given a file or image, converts to ExtFile or ExtImage
vsbabu-removethis@vsbabu.org - 11/25/01
"""
from StringIO import StringIO
#  cStringIO doesn't work! Need to see why
__version__ = '0.0.1'

defmake_stream(o):
    """makes a StringIO stream for a given zodb object o
    
    shameless copy from lib/python/OFS/Image.py , index_html
    """
    (start,end) = (0,o.getSize())
    data = o.data
    if type(data) is type(''): #StringType
        infile = StringIO(data[start:end])
    else:
        pos = 0
        infile = StringIO(data.data)
        while data is not None:
            l =  len(data.data)
            pos = pos + l
            if pos > start:
                # We are within the range
                lstart = l - (pos - start)

                if lstart < 0: lstart = 0
                
                # find the endpoint
                if end <= pos:
                    lend = l - (pos - end)
                    
                    infile.write(data[lstart:lend])
                    break

                # Not yet at the end, transmit what we have.
                infile.write(data[lstart:])

            data = data.next
    infile.seek(0)
    return infile

defzodb2ext(self,o):
    """ Given a file or image, converts to ExtFile or ExtImage """
    
    if o.meta_type == 'File':
        nid = o.getId()
        infile = make_stream(o)
        self.manage_delObjects([nid])
        self.manage_addProduct['ExtFile'].manage_addExtFile(id=nid,
                              title=o.title, descr='', file=infile)
        infile.close()
    elif o.meta_type == 'Image':
        nid = o.getId()
        infile = make_stream(o)
        self.manage_delObjects([nid])
        self.manage_addProduct['ExtFile'].manage_addExtImage(id=nid,
                              title=o.title, descr='', file=infile)
        infile.close()
    else:
        return 'Object should be File or Image'

    return ''
Python Script - convert2ext
from Products.PythonScripts.standard import html_quote
for (name,o) in context.objectItems(['File','Image']):
    context.zodb2ext(o)
    print name

return printed

Long live open source

Quite a few NT administrators are fond of repeating the phrase all that open source crap when your Linux/*BSD machines are happily working and you ask them why they reboot NT servers weekly or why they've a list of mysterious problems - not solutions yet. Somehow getting some courage, I went through Zope's source code and viola, there it was - code for reading the contents of a File/Image object! It was a simple matter to turn it into making a stream.