4.0 - File Storage

IPS News · October 7, 2013

Introduction

The IPS Social Suite needs to store lots of different files - there's attachments and profile photos uploaded by members, CSS and JavaScript files, emoticons, etc.

In IP.Board 3.x, various images got stored in different places:

Files uploaded by users get put in the /uploads directory. If you have a complicated setup, it's difficult to handle these. If you have a load-balanced cluster you need to set up an environment whereby all files are stored on a single server, or all uploaded files are synched between servers, but serving these files over a high-performance CDN can be difficult.
CSS, JavaScript files, images and emoticons get put in /style_* directories. If you want to serve these over a CDN, you can do so, but you need to copy the files over yourself.
Other pieces of data are written to disk as a caching mechanism. This has the same issue with load-balanced environments as file uploads.
Some applications had other methods - for example, IP.Downloads allows you to store files on a remote server using FTP.

In 4.0, we wanted to pull this all together and build a much better system for storing files and build the whole system with high-performance environments in mind.

File Storage

In 4.0, you have several different ways to store files:

On a local server
On a remote server using FTP (which you can use to upload files to many CDN services)
As binary data in the database
On Amazon S3

You can set up different configurations and choose which configuration to use for different types of files. For example, if you want to store user's profile photos on Amazon S3, but you want attachments to be on the local server, or even a different Amazon S3 bucket - 4.0 can handle that. And if at any point you change your mind about which storage method you want to use, the system will automatically handle moving all the files for you.

Everywhere that writes a file will use this central system - so IP.Downloads and IP.Gallery are included too.

Caching

There are lots of places throughout the suite where the same stuff needs to be retrieved or calculated over and over - for example, certain configuration settings, language data, information about the installed applications, etc. If this data can be cached, not only does it alleviate database load, it means the PHP code doesn't need to re-process the data.

In IP.Board 3.x, some of this was stored in a particular database table and could be cached using a proper caching system - but it was difficult to configure, and not everything used it - compiled HTML templates, language strings and more were saved as files in the /cache directory, which causes difficulties for load-balanced cluster environments.

In 4.0, we've overhauled all of this. For things that need cold storage (like compiled HTML templates) - you can choose either the file system or the database for storage. The data can then cached, along with anything else which might benefit from caching (like settings, application data, etc.) using one of 5 supported caching methods: