Tuesday, December 14, 2010

large file cache use mmap

We had a 3G file and fields position can be identified using hash value. We use the mmap to deal with the cache.
(1) open the file
(2) caculate the position, if the data is not in memory, then call:
mmap2(NULL, 16384, PROT_READ|PROT_WRITE, MAP_SHARED, 7, 0x210e8)
which loads the 16K into memory.
(3) for the data in the next request, call the mmap2 again (the mmap can be called multiple times).
(4) When you run out of cache (let's say 300M), find the least used fields, then run
munmap(....)
to cache the fileds out.

Perl provides a module Cache:FastMmap to share memory cross processes.

No comments: