Release Notes For Plasma
This is version: 0.2 "kurzweil". This is an alpha release to make
Plasma known to interested developers.
Changed in 0.2
New features in PlasmaFS:
- New namenode operations:
namelock. Generally, the operations need fewer locks now (less danger
of conflicting changes).
- Namenodes can now resolve symbolic links
Plasma_client API has been extended, e.g. a new
copy_out work now with non-seekable descriptors.
Also, the above mentioned new namenode operations are supported.
plasma client utility
- The NFS bridge support now writes. Also, the mount table is persistent
(restarts of the NFS bridge do not break existing mounts).
Implementation improvements in PlasmaFS:
- Partial rewrite of the
Plasma_client part for buffered I/O
Plasma_client.retry uses a more sophisticated scheme
files database table
New features in the map/reduce framework:
- Files are now processed in chunks of bigblocks, and not only filesystem
blocks. Bigblocks can be larger than blocks.
- Revised configuration scheme. Many parameters can be changed at
job start time.
- Streaming for map/reduce
- User files can be installed on task nodes
- Per-task log files
- Log files are moved to a PlasmaFS directory
Implementation improvements in the map/reduce framework:
- Shuffle tasks are faster when no merging is required
- The number of map tasks is automatically computed.
- Existing PlasmaFS filesystems are incompatible
- There are incompatible protocol changes
What is working and not working in PlasmaFS
Generally, PlasmaFS works as described in the documentation. Crashes
have not been observed for quite some time now, but occasionally one
might see critical exceptions in the log file.
PlasmaFS has so far only been tested on 64 bit, and only on Linux
as operation system. There are known issues for 32 bit machines,
especially the blocksize must not be larger than 4M.
Data safety: Cannot be guaranteed. It is not suggested to put valuable
data into PlasmaFS.
- It is still unclear whether the timeout settings are acceptable.
- There might be name clashes for generated file names. Right now it is
assumed that the random number generator returns unique names, but this
is for sure not the case.
- The generated inode numbers are not necessarily unique after namenode
Not implemented features:
- The namenodes cannot yet detect crashed datanodes. Datanodes are always
reported as alive.
- The ticket system is not fully implemented (support for "read").
- There is no authorization system (file access rights are ignored)
- There is no authentication system to secure filesystem accesses (such
- There are too many hard-coded constants.
- The file name read/lookup functions should never return
errors. (This has been improved in 0.2, though.)
- Translation of access rights to NFS
- Support for checksums
- Support for "host groups", so that it is easier to control which machines
may store which blocks. Semantics have to be specified yet.
- Better block allocation algorithms. The current algorithm works only well
when many blocks are allocated at once. It is very bad when a file is
extended block by block.
- Define how blocks are handled that are allocated but never written.
- Recognition of the death of the coordinator, and restart of the
- Multicast discovery of datanodes
- Lock manager (avoid that clients have to busy wait on locks)
- Restoration of missing replicas
- Rebalancing of the cluster
- Automated copying of the namenode database to freshly added namenode slaves
What is working and not working in Plasma MapReduce
- Management of available RAM is not yet sufficient
Not implemented features:
- Task servers should be able to provide several kinds of jobs
- Think about dynamically extensible task servers
- Run jobs only defining
map but no
- Support for combining (an additional fold function run after each
shuffle task to reduce the amount of data)
- nice web interface
- support user counters as in Hadoop
- restart/relocation of failed tasks
- recompute intermediate files that are no longer accessible due to node
- Speculative execution of tasks
- Support job management (remember which jobs have been run etc.)
- Support to execute several jobs at once
What we will never implement:
- Jobs only consisting of
reduce but no
map cannot be supported
due to the task scheme. (Reason: Input files for sort tasks must