85 lines
4.0 KiB
Markdown
85 lines
4.0 KiB
Markdown
%ExternalStore {#externalstorearch}
|
|
========================
|
|
|
|
%ExternalStore is an optional feature that enables persistent object storage
|
|
outside the main database, primarily for revision text (also known as a "blob").
|
|
|
|
The main public interface for interacting with %ExternalStore is ExternalStoreAccess.
|
|
Though note that higher-level concepts like {@link MediaWiki\Revision\RevisionRecord} and
|
|
text blobs have their own dedicated interface: {@link MediaWiki\Revision\RevisionStore}, and
|
|
{@link MediaWiki\Storage\BlobStore}.
|
|
|
|
## Concepts
|
|
|
|
### URL
|
|
|
|
Objects in external stores are internally identified by a special URL.
|
|
The URL is of the form `<store protocol>://<location>/<object name>`.
|
|
|
|
### Protocol
|
|
|
|
The protocol represents which ExternalStoreMedium class is used. The following protocols are
|
|
supported by default:
|
|
|
|
- `DB`: ExternalStoreDB
|
|
- `http`: ExternalStoreHttp
|
|
- `mwstore`: ExternalStoreMwstore
|
|
|
|
Multiple protocols may be enabled at the same time. For example, to support reading older data
|
|
while using a different protocol for new data.
|
|
|
|
Protocols are configured via {@link $wgExternalStores}. The ExternalStoreMedium class is decided
|
|
based on concatenating the value from $wgExternalStores to the string `ExternalStore`, with a
|
|
ucfirst transformation applied as-needed.
|
|
|
|
A custom protocol called "foobar" could be configured by implementing ExternalStoreMedium in a
|
|
subclass called `ExternalStoreFoobar`.
|
|
|
|
### Location
|
|
|
|
The location identifies a particular instance of given store protocol.
|
|
|
|
In the case of ExternalStoreDB, the location represents a database cluster (one or more database
|
|
servers that hold the same data).
|
|
|
|
When using the default of {@link Wikimedia::Rdbms::LBFactorySimple LBFactorySimple}, these
|
|
clusters can be configured via {@link $wgExternalServers}. Otherwise, external clusters must be
|
|
configured via {@link $wgLBFactoryConf}.
|
|
|
|
## New insertions
|
|
|
|
The destination of newly stored text blobs is configured via {@link $wgDefaultExternalStore}.
|
|
To enable use of %ExternalStore for new blobs, this must be set to a non-empty array. This can
|
|
be disabled to store new blobs in the main database instead, it does not affect how existing
|
|
blobs are read.
|
|
|
|
Each destination uses a partial URL of the form `<store protocol>://<location>`.
|
|
When a blob is inserted, we randomly pick an available protocol/location pair from this list.
|
|
Insertions will fail-over to another default destination if the chosen one is unavailable.
|
|
|
|
## Append-only {#externalstore-appendonly}
|
|
|
|
%ExternalStore is designed as an append-only system, to persist data in a way that is highly
|
|
reliable and immutable. As such, the interface is restricted to fetch and insert operations,
|
|
and specifically does not permit modification or deletion once data is stored.
|
|
|
|
This design benefits MediaWiki in a number of ways:
|
|
|
|
* The limited interface provides flexibility to each protocol implementation.
|
|
* Caching is trivial and safe.
|
|
* Stable references to external store can be kept outside of it, in the core database and anywhere
|
|
else in caching or other storage layers, without needing to track of propagate changes.
|
|
* Historical data can be stored with high reliability guarantees and operational safety:
|
|
* External database clusters may be operated in read-only mode, directly through MySQL.
|
|
* Each replica within the cluster may operate as independent static backup.
|
|
* Database replication between hosts may be turned off.
|
|
* Even command-line access from outside MediaWiki can't accidentally affect historical data.
|
|
|
|
In case of maintenance tasks such as recompression, we generally iterate through known blobs
|
|
and write new blobs as-needed and gracefully update pointers accordingly. If an entire cluster
|
|
has been copied or recompressed to a new location, it can be taken out of rotation, with any
|
|
storage space freed at that time. Note that multiple locations may be physically colocated
|
|
on the same hardware, e.g. by running multiple instances of MySQL. Although it may be simpler
|
|
to free space by doing recompression during other routine maintenance, such as when migrating
|
|
data from old to new hardware.
|