Efficient Algorithm for Deterministic Search of Hot Elements
share
When facing a very large stream of data, it is often desirable to extract most important statistics online in a short time and using small memory. For example, one may want to quickly find the most influential users generating posts online or check if the stream contains many identical elements. In this paper, we study streams containing insertions and deletions of elements from a possibly large set N of size |N| = n, that are being processed by online deterministic algorithms. At any point in the stream the algorithm may be queried to output elements of certain frequency in the already processed stream. More precisely, the most frequent elements in the stream so far. The output is considered correct if the returned elements it contains all elements with frequency greater than a given parameter φ and no element with frequency smaller than φ-ϵ. We present an efficient online deterministic algorithm for solving this problem using O(min(n, polylog(n)/ϵ)) memory and O(polylog(n)) time per processing and outputting an element. It is the first such algorithm as the previous algorithms were either randomized, or processed elements in substantially larger time Ω(min(n, log n/ϵ)), or handled only insertions and required two passes over the stream (i.e., were not truly online). Our solution is almost-optimally scalable (with only a polylogarithmic overhead) and does not require randomness or scanning twice through the stream. We complement the algorithm analysis with a lower bound Ω(min(n, 1/ϵ)) on required memory.
READ FULL TEXT