Configuring the Content Hosting Environment - Glenn Golden
Anytime you want to use the file system feature of content hosting, to store the content hosting resource body binary bytes in files in a file system instead of records in the database, you must a) configure your Sakai to do this, and b) run a conversion to bring your files out of your database and into the file system. This conversion is needed even for a brand new database, since there are some resources shipped with the started Sakai db.
Configuration
The best place for configuring this is the sakai.properties file.
- the file system root for content hosting's external stored files (default is null, i.e. store them in the db)
bodyPath@org.sakaiproject.content.api.ContentHostingService = ${sakai.home}content
Enable the above line, and point at the root folder for the files to be stored.
- when storing content hosting's body bits in files, an optional set of folders just within the content.filesystem.root
- to act as volumes to distribute the files among - a comma separate list of folders. If left out, no volumes will be used.
bodyVolumes@org.sakaiproject.content.api.ContentHostingService = v1,v2,v3
Enable the above line, and set the list of "volumes" for storage. You can specify one or more volume names, comma separated on this line. These are folders under the file system root. Files will be distributed among these volumes.
If you are going to use multiple volume devices, you need to map them to these volume names that live "under" the root. We have done this with our AFS file storage system at the University of Michigan. If you are not using separate devices, then you can use any folder names for the volumes. Provide at least one.
Files will be stored under each volume in a way so that there are not too many in any one folder. The folder structure we use is:
YYYY/DDD/HH/id, where YYYY=year, DDD=day of year, HH=hour of day, and the 1111...=an id-based file name
for example,
2005/070/03/3223479379834-2343
or, using the above root and volumes, it might be:
/usr/local/tomcat/sakai/content/v2/2005/070/03/3223479379834-2343
Note that the resource name and type is not at all encoded here. The date/time used to form the file name is the date/time of file creation.
Conversion
Once you have your database setup, and your file values configured, and the file system read/write accessible to your application server, you need to configure your Sakai to run and convert. Do this once, shutdown after conversion is complete, and re-configure Sakai to no longer do the conversion. At that point you could remove the table CONTENT_RESOURCE_BODY_BINARY.
To convert, add this to your sakai.properties:
convertToFile@org.sakaiproject.content.api.ContentHostingService = true
As soon as Sakai starts up, it will create a file system file for any content hosting resource that has a null "FILE_PATH" field. This could take a long time, if you have many resources. If the process dies mid way, you can restart it, and it will pick up about where it left off (since it does not update the "FILE_PATH" until it has written out the file).
Clustering
If you are running multiple application servers, it is vital that you choose a file system technology that is equally available to all the servers at once. We use our campus AFS system for this. Any network file system should work. Make sure that the file system is mounted in the same place on all the application servers.
The file path is stored in the content hosting resource record. This includes the path from the volume name all the way to the file name. It does NOT include the root. You can change the root location over time, or have it different on each app server, as long as the app server is properly configured. But you can never change the volume names under the root (you could add, but you cannot remove).
John Leasia's Explanation
To point the Resources tool at a particular sub-folder in resources, you can put something like this in the 'home' field of the sakai.resources tool
/group/<siteid>/<subfoldername>/
where siteid is that number that is the sites unique id (something like 2af6cd9b-e3b1-4be7-0041-d2dad816f38e
and subfoldername is the name of the folder.
You need the trailing /
On Configuring to use AFS - Glenn Golden
The code that forms the path is in the content hosting service implementation, either the base or the db version. If you need to change it, you should be able to extend the class and replace the pathing code with your own version, then register the new code as the content hosting service implementation.
But... don't think that you can change it so that you can "tie in" to an existing set of AFS space that the user already has. This will not work. Sakai needs a private file system to store the body bytes of the files in content hosting - this is really a mixed store, with the meta data in the database and the bytes in files in the file system. This private file system must be private to Sakai; the Sakai servers must have credentials to work in there, and no other process should be creating or deleting files from this private file system.
Specifically, if you have end-user AFS space, this cannot be tied into in Sakai with the content hosting service we currently have. Not to say that it can't be written... problems include proxying the end-user's credentials through the Sakai server to access the data, and synchronizing the meta data in the Sakai database with changes made by other applications to the file area.
User Quotas
Below is an email conversation between Paul Beckett and John Leasia on Sakai's ability to define and manage user disk space quotas.
Paul Beckett
I want to give some students larger resources section quotas than others, and would like to remove the resources section from some 'MyWorkspaces' entirely.
Is it possible to remove the 'My Workspace' from some accounts? Is it possible to create or modify 'My Workspace' accounts to have different quotas?
I have a lot of accounts, and would like some automated way of doing this - is it possible in sakaiscript?
John Leasia Replies
If as admin you visit a My Worspace (or any worksite really), then go to the Resources tool there and click Revise Properties, there is a field which can be changed to specify the site quota, and override whatever default you have in place.
I don't know of any way to prevent a user from having a My Workspace. If they don't have one, it is created when they next login.
You can control what is in a particular type of user's my workspace using !user template sites. A user account has a type, and when that account first logs in, it gets a copy of the worksite associated with it's type. The associated worksite it gets is named like !user.<accounttype>. The account type is set when the account is created (the Type field on the account when creating a new user). If there is no !user.<accounttype> site to match the account type of a user, then they get a copy of the !user site for their my workspace. So you can control what tools a particular type of user sees in their my workspace by editing the associated !user template worksite. Once their acount is created though, they have a my workspace of their own and changes to the !user template site wouldn't affect them.
I thought you might be able to create a !user.<accounttype> template site, then edit the site's resources tool and up the quota, and everyone with that account type would then get that new quota, but it didn't seem to work when I tried.
So, those more versed in sakaiscript can respond as to whether you can play with quotas that way.
Paul Beckett replies
I would like to create some different user account types, and modify the tools / quota's for these account types. However I am unclear of where-abouts in the administration worksite I should be making these changes. Sorry for these rather basic questions, but it really isn't clear to me how I should do this, and would really appreciate it if someone could spare the time to point me in the right direction.
Specifically, I would like to:
- create three account types: Undergrad_student , Postgrad_student and Tutor.
- set different default sizes for there workspace quota's, for example Undergrads - 1mb , Postgrads - 2mb and Tutors - 10mb.
- provide different tool lists in the "my workspace", depending on the account type.
- set the resources size of project sites.
Is there a step-by-step guide somewhere that explains how to go about
these tasks?
Also in worksite setup, I cannot see user "My Workspace" sites, clicking on View "My Workspace sites", simply gives a message "No sites were found in the view of "My Workspace", despite their being existing user workspace's (with content).
I'm using Sakai 2.1.0 (maintenance branch: https://source.sakaiproject.org/svn/branches/sakai_2-1/), with MySQL 4.1.
John Leasia replies
As stated below I don't know of a way to selectively set quotas on sites of a particular type other than by running some script.
The different tool lists for different types of accounts can be accomplished by creating site templates for each type of user. !site.template.Undergrad_student, !site.template.Postgrad_student, etc. Edit each of those sites and put in whatever pages/tools you want, and then when a new account logs in, they get a My Workspace according to the associated template. This would affect only accounts which logged in for the first time after you established those templates.
There is no step by step manual at this point. See the ppt presentation made at the Austin conference at http://bugs.sakaiproject.org/confluence/pages/viewpage.action?pageId=9638 for an overview which includes a bit of information about site.templates.
Many have used scripts to do similar things to existing sites using scripts. Maybe someone can supply one and you could set that up as a quartz job I suppose.
File System vs. Database Storage - Charles Severance
You should leave the body path un-set unless your intent is for Sakai to store the blobs in the file system.
I am guessing that you are running this all on a PC as a test - so it makers perfect sense that
${sakai.home} ends up being the path to the tomcat running on your PC.
Storing bodies on the file system is a petty advanced feature. It is in the file system where Tomcat is running - not the filesystem on the mysql server.
On Storing Resources in MySQL
Jim Eng has been working on adding a lot of error checking in content hosting - especially in the store body on disk code - perhaps we need to take patches and merge them into Jim's work on the store bodies in the DB side.
Storing resource bodies in MySql will likely bring asystem to its knees very quickly. Within a few months, you may not even be able to get a backup of MySql to run because it will become so large so quickly.
Adding MIME Types
Supporting Quicktime Moves does not require a streaming server.
Go into
sakai/util/util-util/util/src/java/org/sakaiproject/util/Validator.java/Validator.java
Go to line 701 and add this line
if (lType.equals("video/quicktime")) return true;
Recompile - viola! Quicktime opening in the browser!
Lather, Rinse and Repeat for other mime types that you want in the browser rather than saved to desktop.
Actually we need to make this a configurable property rather than a code change...
