Database File System An Alternative to Hierarchy Based File Systems O Gorter University of Twente August  Database File System — An Alternative to Hierarchy Based File Systems Author: O Gorter Supervisors: H Scholten, B van Dijk, P.G Jansen Copyright C 2003, 2004 O Gorter University of Twente, Computer Sciences Enschede, the Netherlands http://ozy.student.utwente.nl Database File System Abstract —— Over 30 years people worked with hierarchical file systems; file systems based on directories and files These file systems have proven their use, but today, with gigabytes of storage and millions of files, managing files with directories is becoming increasingly difficult This research presents an alternative to hierarchy based file systems At the basis of this system is a unified approach to all properties of a file Combining this approach with a drag and drop user interface creates an alternative that is as usable as directories, while in the mean time going well beyond the expressive powers found in hierarchy based file systems The end result is a file system which makes working with files a lot easier on the user Emphasis of this systems lies at the user and not the operating system, therefor files like shared libraries not show up in this system, these should be stored by other means The implementation is an abstraction layer above a hierarchy based file system and the two work together in such a way that a high level of backwards compatibility is achieved, not rendering current programs useless This setup has been tested on users and the results indicate that this system is a very valid alternative to hierarchical file systems O Gorter III Database File System Foreword —— This document is my graduation report for the University of Twente and is all about the Database File System (dbfs); my graduation project Because this document is mainly written for the University some content might not be relevant to all readers For those only interested in what the dbfs is, I recommend reading chapters “Introduction” and “Database File System – Overview” The document assumes the reader has a fair knowledge of current computer systems and related terms As a reference, most terms are explained in the chapter “List of Used Terms and Abbreviations”, related research and documents are in chapter “References” All of the research in this document is done from a user and user interface point of view This is different from most research on file systems, and explains why compared to these writings, seemingly important information is left out, while almost trivial points are discussed in depth For those who obtained this document digitally, it is available in two versions, one optimized for screen reading and one optimized for print (dbfs-screen.pdf and dbfs-paper.pdf resp.) The screen version has a : layout with a slightly larger font and is definitely recommended when reading from a monitor Thanks For all their advice, support, help and faith, I would like to thank Hans Scholten; Betsy van Dijk; Pierre Jansen; all the others on the dies group; my flatmates; and my family Also a special thanks goes out to all the usability testers, whom I will only mention by first name, but you know who you are O Gorter IV Database File System Albert Bas Coen Edgar Epco Erwin O Gorter Foreword Gerben Harry Igor Ivo Janneke Joris Jeshua Marc Martin Stefan Tim Rutger V Database File System Contents —— Abstract III Foreword IV  Introduction . Relevance    Hierachy Based File Systems – Theory . Properties of Hierarchy Based File Systems    Database File System – Theory . Relevance  . Categorisation    Database File System – Overview . User Interaction    Database . . .   File System – Internals Server  Client  Graphical User Interface Usability Testing O Gorter   VI Database File System . . .  Contents Objective  Method  Results  Conclusions . Future Work and Recommendations   References  Related Work  Related Software  Other References  Index  List of Used Terms and Abbreviations  A Assignment – Database File System  B Dutch Abstract  C Test – Email  D Test – Arrange E Interview – Email Normal  F Interview – Email Dbfs  G Interview – Arrange  H DBFS Source File Listing  O Gorter  VII Database File System Introduction —1— File access and file management is something we on our pcs every day; lots of computer time is spend browsing our directories and opening and saving files, or worse, finding files The basis for this system was laid down over 30 years ago, and since graphical user interfaces became main-stream not much has changed Yet computer hardware has become increasingly more powerful and limitations that existed are no longer there Still the biggest change in our file interface is preview (thumbnail) rendering in the file manager A new file system can introduce better metaphors on working with files and can make use of advanced gui techniques not available when hierarchical file systems came into use It can bring the focus of the file system to the user, instead of the computer, and in doing so change how we think about files and the whole computer It can be an enabler for a new and more up-to-date user-oriented computer interface And this is exactly what this research is about, trying to bring file management to the user It does so by providing a search based file interface, based on file meta-data, and it introduces keywords in favor off directories Being user oriented means only storing documents and not system files like shared libraries Where documents are all files the user is interested in, this can be a msword document, but also images, music and more Because the systems searches and modifies meta-data, all meta-data can be treated equally, meaning that security, ownership and sharing are just as easy to manipulate as the file-name or keywords And without directories the systems does away with locations, instead it can categorize documents in a more powerful way Without locations on a file system, we can have applications that just save everything you The save button can be completely removed from every interface element of the computer, doing away with the dualistic nature O Gorter  Database File System Introduction of how we use a computer today; creating a system where there is no longer a difference on what you see on your screen and what is stored on the hard disk . Relevance In the “References” we see more works that try to extend the file system with searching And it is a very relevant idea, because lately it is sometimes easier to find things on the enormous internet, using google, then on your own hard drive You can always use a search, but it is slow and not very ubiquitous This is known by the two mayor os software vendors, Microsoft and Apple: both announced a more integrated search using some sort of database But we will have to wait a while until the new products are available But already we can see what a non-hierarchical approach to files can when we look at specialized applications like iPhoto or Adobe Photoshop Album for digital photo management, or iTunes for digital music O Gorter  Database File System Hierachy Based File Systems – Theory —2— This is chapter is an overview of how todays (hierarchical) file systems work from a high level point of view Where appropriate some forward references are given to the database file system Four properties of hierarchy based filesystems are discussed, with an emphasis on their weak points But we start with a description of what a hierarchy based filesystem actually is Hierarchy based filesystems are created by directories and files A directory is an object wich has a name and it can contain directories and files Files are objects that also have a name and contain data This data can be anything and is only relevant for the application that uses this data; the file system does not impose what the data must look like, in fact, this data is not shown in the file system We use a file system in order to keep track of this data, by keeping track of our files We keep track of our files by knowing their name and the directory in wich they reside This type of filesystem creates a two dimensional space laid out as a tree like structure This structure is created from directory names and depth of directories (Also see Figure 2.1.) By choosing useful names for directories, we effectively create a categorisation over sub-hierarchies (sub-trees) and the files they contain For example, a digital picture from a certain user on a MS Windows system would typically be found somewhere below: /Documents and Settings/username/My Pictures/ We can trace back why the picture would be stored there: The picture is a document, therefor it should be in Documents and Settings; The picture is from ’username’ therefor it should be in username; and it should be in My Pictures because it is a picture O Gorter  Database File System How fast or slow you think you completed the test? slow fast If you had to the test again, you feel you would be faster or slower? slower Interview – Arrange faster Where there things you disliked about the interface, like certain buttons, or how you had to move stuff around? No / Yes, namely: 10 When you had to categorize the pictures, at first, did you know what to do? not clear at all 11 very clear What did you to categorize the pictures? added keyword(s) renamed some/all of the pictures added a custom stored search other: O Gorter  Database File System 12 Interview – Arrange Did you use multiple keywords on a file? No / Yes 13 If you had to the test again, would you use a different approach to categorization? Same / Different 14 When you had to look up the pictures, did the system work as you expected? not at all 15 Do think you performed the asked tasks fast or slow? slow 16 yes fast To select the asked pictures, what features did you use: main types (Documents, Images, ) keywords search field date search custom stored search other: O Gorter  Database File System 17 Interview – Arrange Was there a set of foto’s you could not create using the system? No / Yes, namely: 18 The system does not use directories Without them, you think you can still sufficiently arrange your files (by using keywords or other properties)? No 19 Yes If this type of file access was built into the system you use today (into Windows for example), and you had to use it to manage all your files, would you like that? not at all 20 very much If you have any thoughts on the following items, or something else, please write them down below • • • • No directories, no locations, no organization? Keywords can be like playlists? What if you could use more properties like music file information artiest, album, year, ? Other properties you can add to files (just like keywords), like owners, encryption, ? O Gorter  Database File System Interview – Arrange Thank You O Gorter  Database File System DBFS Source File Listing —H—  The README file Database File System author: Onne Gorter * ABOUT * This is my Master Thesis work for the University of Twente My thesis is about a new file system and interface to this file system The new system is databased, henceforth the name dbfs: database file system * WHAT IS HERE * README: this file university: my thesis report and presentations and other stuff ocamldbfs: implementation of the base system; a server; a client; an ocaml library; a c library and other libraries qtdbfs: implementation of GUI using only qt (incomplete) O Gorter  Database File System DBFS Source File Listing kdelibs: all the changed files in kdelibs (version 3.2.3) to implement dbfs in KDE completely Thus replacing the file-open, file-save dialogs kdbfs: implementation of GUI interface from KDE Needs kdelibs macdbfs: some work at an implementation of a GUI for Mac OS X (Cocoa, inclomplete) dbfsmplayer: a Mac OS X GUI for mplayer (media player) which finds movies and music on its own (Cocoa) Makefile: makes everything from documentation to sources, but not very usefull except for myself The complete archive is managed by subversion version control system (http://subversion.tigris.org) just in case some ’.svn/’ directories show up Note that this can give some trouble when using (an old version of) Interface Builder on Mac OS X * INSTALL * In order to install dbfs you need to have (installed): sqlite: http://www.sqlite.org/ ocaml: http://caml.inria.fr/ qt 3.x or better (optional): http://www.trolltech.com/ KDE 3.2.3 or better (optional): http://www.kde.org/ Mac OS X 10.3 or better (optional): http://www.apple.com/ There are two parts to my project, a server and some client implementations In order to work with the clients you need the server compiled and installed, which provides a set of shared libraries the clients need O Gorter  Database File System DBFS Source File Listing * compiling the server * cd ocamldbfs make make install If ’make’ fails, edit ’Makefile.inc’ to reflect your setup, the makefiles assume that the dependend libraries have been installed with a prefix of ’/usr/local’ The target of the makefiles is ’$HOME’ Change all this in ’Makefile.inc’ and run ’make clean’ ’make’ ’make install’ Edit ’Makefile’ if you want to change the support libraries that get installed If ’make install’ fails check if you need to be superuser You are left with: /usr/local/bin/dbfsd: the server daemon /usr/local/bin/dbfsnotify: a client to make daemon sync with disk /usr/local/include/dbfs.h: client library headers /usr/local/lib/libdbfs.so: client library /usr/local/share/dbfs/dbfs.conf: a default configuration /usr/local/share/dbfs/mime.conf: a default mime type map client/client: a very basic client written in ocaml c-api/example: a very basic c implemented client, check ’c-api/main.c’ for its sources There are also objective-c and C++ libraries available and some clients need these If all goes well they are installed as appropriate for your system otherwise install them by hand: O Gorter  Database File System cd objc-api make make install DBFS Source File Listing (or cpp-api) this will give you: /usr/local/lib/libdbfs_objc.so /usr/local/include/dbfs_objc.h Again, ’make install’ probably needs superuser powers Note that the objective-c library needs Foundation classes from the Mac OS X or GNUstep (OpenStep) environment, the library has only been tested for Mac OS X * compiling the clients * If you are under KDE you can compile a kde client, which is a pretty usable client Qt needs to be installed with STL! this is not default for KDE with most distributions unfortunately cd kdbfs qmake make /kdbfs If under Mac OS X you can compile a media player based on MPlayer OS X, you need the developers tools (XCode) to be installed Also the objective-c library needs to be installed, see above cd dbfsmplayer open dbfs\ MPlayer.pbproj and use XCode to compile and run O Gorter  Database File System DBFS Source File Listing Most platforms can run the qt client C++ library needs to be installed The Qt client is not at all up to date cd qtdbfs qmake make /qtdbfs (or under macosx: ’open qtdbfs.app/’) * WHOLE KDE SETUP * If you want to install a whole new KDE environment which works with only dbfs (ie no more normal file access) This has been done on x86 linux, the following steps are needed to reproduce that: download to the directory ~/Sources/kde/ * * * * * * * qt-x11-free-3.3.2.tar.bz2 arts-1.2.3.tar.bz2 kdelibs-3.2.3.tar.bz2 kdebase-3.2.3.tar.bz2 kdeartwork-3.2.3.tar.bz2 koffice-1.3.1.tar.bz2 (optional) kdevelop-3.0.4.tar.bz2 (optional) setup your environment to use our QTDIR and KDEDIR: (in bashrc) export QTDIR="$HOME/qt-testing" export KDEDIR="$HOME/kde-testing" export PATH="$QTDIR/bin:$KDEDIR/bin:$HOME/bin:$PATH" export LD_LIBRARY_PATH="$QTDIR/lib:$KDEDIR/lib:$HOME/lib:$LD_LIBRARY_PATH" export KDEHOME="$HOME/.kde-home-testing" make sure your environment is correctly setup in the current shell! O Gorter  Database File System DBFS Source File Listing tar -xjvf qt-x11-free-3.3.2.tar.bz2 cd ln -sf Sources/kde/qt-x11-free-3.3.2 qt-testing mkdir kde-testing mkdir kde-home-testing cd Sources/kde/qt-x11-free-3.3.2 /configure -system-zlib -qt-gif -system-libpng \ -system-libjpeg -plugin-imgfmt-mng -thread \ -no-xinerama -no-g++-exceptions (accept the license by typing ’yes’ ) make cd (NO make install !!) tar -xjvf arts-1.2.3.tar.bz2 cd arts-1.2.3 /configure with-extra-includes=$HOME/include with-extra-libs=$home/lib make make install tar -xjvf kdelibs-3.2.3.tar.bz2 cd kdelibs-3.2.3 cp -Rf /path/to/development-tree-dbfs/kdelibs/* sh admin/cvs.sh cvs /configure with-extra-includes=$HOME/include with-extra-libs=$HOME/lib make make install Repeat the arts steps for each other package: O Gorter  Database File System DBFS Source File Listing * kdebase-3.2.3 * kdeartwork-3.2.3 (optional) * koffice-1.3.1 (optional) To install the main interface: cd /path/to/development-tree-dbfs/kdbfs qmake make Done? good, now we setup some final settings: cd cp -Rf /path/to/development-tree-dbfs/share ~/.kde-home-testing/ mkdir Applications mkdir Documents mkdir Misc mkdir Movies mkdir Music mkdir Pictures cd cp cd cp kde-testing/share/applnk ~/Applications /applications/kde/ ~/Applications finally setup done To run KDE use startkde or place startkde in xinitrc (exec startkde) or in Xclients for redhat users But first start the dbfs daemon following the next section * RUNNING AND USING * O Gorter  Database File System DBFS Source File Listing You need to create a directory ’~/.dbfs/’ and copy ’/usr/local/share/dbfs/*’ to this directory Edit ’~/.dbfs/dbfs.conf’ (This could be in ~/share/dbfs/) To run a client first run the server then the client, run the server with ’dbfsd’ Use ’dbfsd help’ to read about its options Note that the server is personal, and not run as superuser With the current implementation you cannot run a server for multiple users (but each one can run its own private version)  dbfsd Directory Listing ocamldbfs/c-api/customcaml.ml ocamldbfs/c-api/filecaml.ml ocamldbfs/c-api/filtercaml.ml ocamldbfs/c-api/keywordcaml.ml ocamldbfs/c-api/viewcaml.ml ocamldbfs/client/client.ml ocamldbfs/client/customs.ml ocamldbfs/client/dbfsnotify.ml ocamldbfs/client/files.ml ocamldbfs/client/init.ml ocamldbfs/client/keywords.ml ocamldbfs/client/view.ml ocamldbfs/common/config.ml ocamldbfs/common/custom.ml ocamldbfs/common/file.ml ocamldbfs/common/filter.ml ocamldbfs/common/keyword.ml O Gorter  Database File System DBFS Source File Listing ocamldbfs/common/log.ml ocamldbfs/common/mime.ml ocamldbfs/common/observer.ml ocamldbfs/common/protocol.ml ocamldbfs/common/stringextra.ml ocamldbfs/common/threadobjects.ml ocamldbfs/server/crawler.ml ocamldbfs/server/database.ml ocamldbfs/server/files.ml ocamldbfs/server/init.ml ocamldbfs/server/mimedatabase.ml ocamldbfs/server/server.ml ocamldbfs/server/view.ml ocamldbfs/sqlite/sqlite.ml  kdelibs Directory Listing kdelibs/kdeui/klistview.cpp kdelibs/kio/dbfstests/kfiledialogtest.cpp kdelibs/kio/dbfstests/kfiledialogtest.h kdelibs/kio/kfile/customwidget.cpp kdelibs/kio/kfile/customwidget.h kdelibs/kio/kfile/datewidget.cpp kdelibs/kio/kfile/datewidget.h kdelibs/kio/kfile/dbfsfileiconview.cpp kdelibs/kio/kfile/dbfsfileiconview.h kdelibs/kio/kfile/dbfshandler.cpp kdelibs/kio/kfile/dbfshandler.h kdelibs/kio/kfile/filterwidget.cpp O Gorter  Database File System DBFS Source File Listing kdelibs/kio/kfile/filterwidget.h kdelibs/kio/kfile/kdbfsdialog.cpp kdelibs/kio/kfile/kdbfsdialog.h kdelibs/kio/kfile/kencodingfiledialog.cpp kdelibs/kio/kfile/kencodingfiledialog.h kdelibs/kio/kfile/keywordwidget.cpp kdelibs/kio/kfile/keywordwidget.h kdelibs/kio/kfile/kfiledialog.cpp kdelibs/kio/kfile/kfiledialog.h kdelibs/kio/kfile/kfileiconview.cpp kdelibs/kio/kfile/kfileiconview.h kdelibs/kio/kfile/searchwidget.cpp kdelibs/kio/kfile/searchwidget.h kdelibs/kio/kfile/sizewidget.cpp kdelibs/kio/kfile/sizewidget.h kdelibs/kio/kfile/switcher.cpp kdelibs/kio/kfile/switcher.h kdelibs/kio/kfile/viewwidget.cpp kdelibs/kio/kfile/viewwidget.h kdelibs/kio/kio/job.cpp O Gorter  Database File System O Gorter DBFS Source File Listing  ... is Hierarchy based filesystems are created by directories and files A directory is an object wich has a name and it can contain directories and files Files are objects that also have a name and... Hierachy Based File Systems – Theory Properties of Hierarchy Based File Systems URLs Using directories and files, hierarchy based file systems create a unique name for every file; referred to as the... simultaneous access to the database can lead to incorrect results And simultaneous access to the sending mechanisms can lead to incorrect data being transmitted We also see constructs like debug ~file: "database. ml"