In the book I'm using for my class (and from what I've seen from a few other places), it seems like the algorithm for creating a huffman tree stems from
(1) Building a minheap based on the frequency of each character in whatever file or string is being read in.
(2) Popping off the 2 smallest values from the minheap and combining their weights into a new node.
(3) Re-inserting the new node back into the same minheap.
I'm confused about step 3. Most huffman trees I've seen have attributes more similar to a max heap than a minheap (although they are not complete trees). That is to say, the root contains the maximum weight (or combination of weights rather), while all of it's children have lesser weights. How does this implementation give a huffman tree when the combined nodes are put back into a minheap? I've been struggling with this for a while now.
A similar question has already been posted here (with the same book as me): I don't understand this Huffman algorithm implementation
In case you wanted to see the exact function described in (3).
Thanks for any help!
A Huffman tree is often not a complete binary tree, and so is not a min-heap.
The Huffman algorithm is easily understood as a list of frequencies from which a tree is built. Small branches are constructed first, which will eventually all be merged into a single tree. Each list item starts off as a symbol, and later may be a symbol or a sub-tree that has been built. Each list item always has a frequency (an integer count usually).
Take the two smallest frequencies out of the list (ties don't matter -- any choice will result in an optimal code, though there may be more than one optimal code). Construct a single-level binary tree from those two, where the two leaves are the symbols for those frequencies. Add the frequencies to make a new frequency representing the tree. Put that frequency back in the list. The list now has one less frequency in it.
Repeat. Now the binary tree constructed at each step may have symbol leaves on each branch, or one leaf and a previously constructed tree, or two trees (at earliest in the third step).
Keep going until there is only one frequency left in the list. That will be the sum of all the original frequencies. That frequency has the complete Huffman tree associated with it.
Now you can (arbitrarily) assign a 0 and a 1 to each binary branch. You build codes or decode codes by traversing the tree from the root to a symbol. The bits from the branches of that traverse are in order the Huffman code for that symbol.
I am wondering what the particular applications of binary trees are. Could you give some real examples?
To squabble about the performance of binary-trees is meaningless - they are not a data structure, but a family of data structures, all with different performance characteristics. While it is true that unbalanced binary trees perform much worse than self-balancing binary trees for searching, there are many binary trees (such as binary tries) for which "balancing" has no meaning.
Applications of binary trees
Binary Search Tree - Used in many search applications where data is constantly entering/leaving, such as the map and set objects in many languages' libraries.
Binary Space Partition - Used in almost every 3D video game to determine what objects need to be rendered.
Binary Tries - Used in almost every high-bandwidth router for storing router-tables.
Hash Trees - Used in torrents and specialized image-signatures in which a hash needs to be verified, but the whole file is not available. Also used in blockchains for eg. Bitcoin.
Heaps - Used in implementing efficient priority-queues, which in turn are used for scheduling processes in many operating systems, Quality-of-Service in routers, and A* (path-finding algorithm used in AI applications, including robotics and video games). Also used in heap-sort.
Huffman Coding Tree (Chip Uni) - Used in compression algorithms, such as those used by the .jpeg and .mp3 file-formats.
GGM Trees - Used in cryptographic applications to generate a tree of pseudo-random numbers.
Syntax Tree - Constructed by compilers and (implicitly) calculators to parse expressions.
Treap - Randomized data structure used in wireless networking and memory allocation.
T-tree - Though most databases use some form of B-tree to store data on the drive, databases which keep all (most) their data in memory often use T-trees to do so.
The reason that binary trees are used more often than n-ary trees for searching is that n-ary trees are more complex, but usually provide no real speed advantage.
In a (balanced) binary tree with m nodes, moving from one level to the next requires one comparison, and there are log_2(m) levels, for a total of log_2(m) comparisons.
In contrast, an n-ary tree will require log_2(n) comparisons (using a binary search) to move to the next level. Since there are log_n(m) total levels, the search will require log_2(n)*log_n(m) = log_2(m) comparisons total. So, though n-ary trees are more complex, they provide no advantage in terms of total comparisons necessary.
(However, n-ary trees are still useful in niche-situations. The examples that come immediately to mind are quad-trees and other space-partitioning trees, where divisioning space using only two nodes per level would make the logic unnecessarily complex; and B-trees used in many databases, where the limiting factor is not how many comparisons are done at each level but how many nodes can be loaded from the hard-drive at once)
When most people talk about binary trees, they're more often than not thinking about binary search trees, so I'll cover that first.
A non-balanced binary search tree is actually useful for little more than educating students about data structures. That's because, unless the data is coming in in a relatively random order, the tree can easily degenerate into its worst-case form, which is a linked list, since simple binary trees are not balanced.
A good case in point: I once had to fix some software which loaded its data into a binary tree for manipulation and searching. It wrote the data out in sorted form:
Alice
Bob
Chloe
David
Edwina
Frank
so that, when reading it back in, ended up with the following tree:
Alice
/ \
= Bob
/ \
= Chloe
/ \
= David
/ \
= Edwina
/ \
= Frank
/ \
= =
which is the degenerate form. If you go looking for Frank in that tree, you'll have to search all six nodes before you find him.
Binary trees become truly useful for searching when you balance them. This involves rotating sub-trees through their root node so that the height difference between any two sub-trees is less than or equal to 1. Adding those names above one at a time into a balanced tree would give you the following sequence:
1. Alice
/ \
= =
2. Alice
/ \
= Bob
/ \
= =
3. Bob
_/ \_
Alice Chloe
/ \ / \
= = = =
4. Bob
_/ \_
Alice Chloe
/ \ / \
= = = David
/ \
= =
5. Bob
____/ \____
Alice David
/ \ / \
= = Chloe Edwina
/ \ / \
= = = =
6. Chloe
___/ \___
Bob Edwina
/ \ / \
Alice = David Frank
/ \ / \ / \
= = = = = =
You can actually see whole sub-trees rotating to the left (in steps 3 and 6) as the entries are added and this gives you a balanced binary tree in which the worst case lookup is O(log N) rather than the O(N) that the degenerate form gives. At no point does the highest NULL (=) differ from the lowest by more than one level. And, in the final tree above, you can find Frank by only looking at three nodes (Chloe, Edwina and, finally, Frank).
Of course, they can become even more useful when you make them balanced multi-way trees rather than binary trees. That means that each node holds more than one item (technically, they hold N items and N+1 pointers, a binary tree being a special case of a 1-way multi-way tree, with 1 item and 2 pointers).
With a three-way tree, you end up with:
Alice Bob Chloe
/ | | \
= = = David Edwina Frank
/ | | \
= = = =
This is typically used in maintaining keys for an index of items. I've written database software optimised for the hardware where a node is exactly the size of a disk block (say, 512 bytes) and you put as many keys as you can into a single node. The pointers in this case were actually record numbers into a fixed-length-record direct-access file separate from the index file (so record number X could be found by just seeking to X * record_length).
For example, if the pointers are 4 bytes and the key size is 10, the number of keys in a 512-byte node is 36. That's 36 keys (360 bytes) and 37 pointers (148 bytes) for a total of 508 bytes with 4 bytes wasted per node.
The use of multi-way keys introduces the complexity of a two-phase search (multi-way search to find the correct node combined with a small sequential (or linear binary) search to find the correct key in the node) but the advantage in doing less disk I/O more than makes up for this.
I see no reason to do this for an in-memory structure, you'd be better off sticking with a balanced binary tree and keeping your code simple.
Also keep in mind that the advantages of O(log N) over O(N) don't really appear when your data sets are small. If you're using a multi-way tree to store the fifteen people in your address book, it's probably overkill. The advantages come when you're storing something like every order from your hundred thousand customers over the last ten years.
The whole point of big-O notation is to indicate what happens as the N approaches infinity. Some people may disagree but it's even okay to use bubble sort if you're sure the data sets will stay below a certain size, as long as nothing else is readily available :-)
As to other uses for binary trees, there are a great many, such as:
Binary heaps where higher keys are above or equal to lower ones rather than to the left of (or below or equal to and right);
Hash trees, similar to hash tables;
Abstract syntax trees for compilation of computer languages;
Huffman trees for compression of data;
Routing trees for network traffic.
Given how much explanation I generated for the search trees, I'm reticent to go into a lot of detail on the others, but that should be enough to research them, should you desire.
The organization of Morse code is a binary tree.
A binary tree is a tree data structure in which each node has at most two child nodes, usually distinguished as "left" and "right". Nodes with children are parent nodes, and child nodes may contain references to their parents. Outside the tree, there is often a reference to the "root" node (the ancestor of all nodes), if it exists. Any node in the data structure can be reached by starting at root node and repeatedly following references to either the left or right child. In a binary tree a degree of every node is maximum two.
Binary trees are useful, because as you can see in the picture, if you want to find any node in the tree, you only have to look a maximum of 6 times. If you wanted to search for node 24, for example, you would start at the root.
The root has a value of 31, which is greater than 24, so you go to the left node.
The left node has a value of 15, which is less than 24, so you go to the right node.
The right node has a value of 23, which is less than 24, so you go to the right node.
The right node has a value of 27, which is greater than 24, so you go to the left node.
The left node has a value of 25, which is greater than 24, so you go to the left node.
The node has a value of 24, which is the key we are looking for.
This search is illustrated below:
You can see that you can exclude half of the nodes of the entire tree on the first pass. and half of the left subtree on the second. This makes for very effective searches. If this was done on 4 billion elements, you would only have to search a maximum of 32 times. Therefore, the more elements contained in the tree, the more efficient your search can be.
Deletions can become complex. If the node has 0 or 1 child, then it's simply a matter of moving some pointers to exclude the one to be deleted. However, you can not easily delete a node with 2 children. So we take a short cut. Let's say we wanted to delete node 19.
Since trying to determine where to move the left and right pointers to is not easy, we find one to substitute it with. We go to the left sub-tree, and go as far right as we can go. This gives us the next greatest value of the node we want to delete.
Now we copy all of 18's contents, except for the left and right pointers, and delete the original 18 node.
To create these images, I implemented an AVL tree, a self balancing tree, so that at any point in time, the tree has at most one level of difference between the leaf nodes (nodes with no children). This keeps the tree from becoming skewed and maintains the maximum O(log n) search time, with the cost of a little more time required for insertions and deletions.
Here is a sample showing how my AVL tree has kept itself as compact and balanced as possible.
In a sorted array, lookups would still take O(log(n)), just like a tree, but random insertion and removal would take O(n) instead of the tree's O(log(n)). Some STL containers use these performance characteristics to their advantage so insertion and removal times take a maximum of O(log n), which is very fast. Some of these containers are map, multimap, set, and multiset.
Example code for an AVL tree can be found at http://ideone.com/MheW8
The main application is binary search trees. These are a data structure in which searching, insertion, and removal are all very fast (about log(n) operations)
One interesting example of a binary tree that hasn't been mentioned is that of a recursively evaluated mathematical expression. It's basically useless from a practical standpoint, but it is an interesting way to think of such expressions.
Basically each node of the tree has a value that is either inherent to itself or is evaluated by recursively by operating on the values of its children.
For example, the expression (1+3)*2 can be expressed as:
*
/ \
+ 2
/ \
1 3
To evaluate the expression, we ask for the value of the parent. This node in turn gets its values from its children, a plus operator and a node that simply contains '2'. The plus operator in turn gets its values from children with values '1' and '3' and adds them, returning 4 to the multiplication node which returns 8.
This use of a binary tree is akin to reverse polish notation in a sense, in that the order in which operations are performed is identical. Also one thing to note is that it doesn't necessarily have to be a binary tree, it's just that most commonly used operators are binary. At its most basic level, the binary tree here is in fact just a very simple purely functional programming language.
Applications of Binary tree:
Implementing routing table in router.
Data compression code
Implementation of Expression parsers and expression solvers
To solve database problem such as indexing.
Expression evaluation
I dont think there is any use for "pure" binary trees. (except for educational purposes)
Balanced binary trees, such as Red-Black trees or AVL trees are much more useful, because they guarantee O(logn) operations. Normal binary trees may end up being a list (or almost list) and are not really useful in applications using much data.
Balanced trees are often used for implementing maps or sets.
They can also be used for sorting in O(nlogn), even tho there exist better ways to do it.
Also for searching/inserting/deleting Hash tables can be used, which usually have better performance than binary search trees (balanced or not).
An application where (balanced) binary search trees would be useful would be if searching/inserting/deleting and sorting would be needed. Sort could be in-place (almost, ignoring the stack space needed for the recursion), given a ready build balanced tree. It still would be O(nlogn) but with a smaller constant factor and no extra space needed (except for the new array, assuming the data has to be put into an array). Hash tables on the other hand can not be sorted (at least not directly).
Maybe they are also useful in some sophisticated algorithms for doing something, but tbh nothing comes to my mind. If i find more i will edit my post.
Other trees like f.e. B+trees are widely used in databases
Binary trees are used in Huffman coding, which are used as a compression code.
Binary trees are used in Binary search trees, which are useful for maintaining records of data without much extra space.
One of the most common application is to efficiently store data in sorted form in order to access and search stored elements quickly. For instance, std::map or std::set in C++ Standard Library.
Binary tree as data structure is useful for various implementations of expression parsers and expression solvers.
It may also be used to solve some of database problems, for example, indexing.
Generally, binary tree is a general concept of particular tree-based data structure and various specific types of binary trees can be constructed with different properties.
In C++ STL, and many other standard libraries in other languages, like Java and C#. Binary search trees are used to implement set and map.
One of the most important application of binary trees are balanced binary search trees like:
Red-Black trees
AVL trees
Scapegoat trees
These type of trees have the property that the difference in heights of left subtree and right subtree is maintained small by doing operations like rotations each time a node is inserted or deleted.
Due to this, the overall height of the tree remains of the order of log n and the operations such as search, insertion and deletion of the nodes are performed in O(log n) time. The STL of C++ also implements these trees in the form of sets and maps.
They can be used as a quick way to sort data. Insert data into a binary search tree at O(log(n)). Then traverse the tree in order to sort them.
Implementations of java.util.Set
On modern hardware, a binary tree is nearly always suboptimal due to bad cache and space behaviour. This also goes for the (semi)balanced variants. If you find them, it is where performance doesn't count (or is dominated by the compare function), or more likely for historic or ignorance reasons.
your programs syntax, or for that matter many other things such as natural languages can be parsed using binary tree (though not necessarily).
BST a kind of binary tree is used in Unix kernels for managing a set of virtual memory areas(VMAs).
Nearly all database (and database-like) programs use a binary tree to implement their indexing systems.
A compiler who uses a binary tree for a representation of a AST, can use known algorithms for
parsing the tree like postorder,inorder.The programmer does not need to come up with it's own algorithm.
Because a binary tree for a source file is higher than the n-ary tree,it's building takes more time.
Take this production:
selstmnt := "if" "(" expr ")" stmnt "ELSE" stmnt
In a binary tree it will have 3levels of nodes, but the n-ary tree will have 1 level(of chids)
That's why Unix based OS-s are slow.
Why is std::map implemented as a red-black tree?
There are several balanced binary search trees (BSTs) out there. What were design trade-offs in choosing a red-black tree?
Probably the two most common self balancing tree algorithms are Red-Black trees and AVL trees. To balance the tree after an insertion/update both algorithms use the notion of rotations where the nodes of the tree are rotated to perform the re-balancing.
While in both algorithms the insert/delete operations are O(log n), in the case of Red-Black tree re-balancing rotation is an O(1) operation while with AVL this is a O(log n) operation, making the Red-Black tree more efficient in this aspect of the re-balancing stage and one of the possible reasons that it is more commonly used.
Red-Black trees are used in most collection libraries, including the offerings from Java and Microsoft .NET Framework.
It really depends on the usage. AVL tree usually has more rotations of rebalancing. So if your application doesn't have too many insertion and deletion operations, but weights heavily on searching, then AVL tree probably is a good choice.
std::map uses Red-Black tree as it gets a reasonable trade-off between the speed of node insertion/deletion and searching.
The previous answers only address tree alternatives and red black probably only remains for historical reasons.
Why not a hash table?
A type only requires < operator (comparison) to be used as a key in a tree. However, hash tables require that each key type has a hash function defined. Keeping type requirements to a minimum is very important for generic programming so you can use it with a wide variety of types and algorithms.
Designing a good hash table requires intimate knowledge of the context it which it will be used. Should it use open addressing, or linked chaining? What levels of load should it accept before resizing? Should it use an expensive hash that avoids collisions, or one that is rough and fast?
Since the STL can't anticipate which is the best choice for your application, the default needs to be more flexible. Trees "just work" and scale nicely.
(C++11 did add hash tables with unordered_map. You can see from the documentation it requires setting policies to configure many of these options.)
What about other trees?
Red Black trees offer fast lookup and are self balancing, unlike BSTs. Another user pointed out its advantages over the self-balancing AVL tree.
Alexander Stepanov (The creator of STL) said that he would use a B* Tree instead of a Red-Black tree if he wrote std::map again, because it is more friendly for modern memory caches.
One of the biggest changes since then has been the growth of caches.
Cache misses are very costly, so locality of reference is much more
important now. Node-based data structures, which have low locality of
reference, make much less sense. If I were designing STL today, I
would have a different set of containers. For example, an in-memory
B*-tree is a far better choice than a red-black tree for implementing
an associative container. - Alexander Stepanov
Should maps always use trees?
Another possible maps implementation would be a sorted vector (insertion sort) and binary search. This would work well
for containers which aren't modified often but are queried frequently.
I often do this in C as qsort and bsearch are built in.
Do I even need to use map?
Cache considerations mean it rarely makes sense to use std::list or std::deque over std:vector even for those situations we were taught in school (such as removing an element from the middle of the list).
Applying that same reasoning, using a for loop to linear search a list is often more efficient and cleaner than building a map for a few lookups.
Of course choosing a readable container is usually more important than performance.
AVL trees have a maximum height of 1.44logn, while RB trees have a maximum of 2logn. Inserting an element in a AVL may imply a rebalance at one point in the tree. The rebalancing finishes the insertion. After insertion of a new leaf, updating the ancestors of that leaf has to be done up to the root, or up to a point where the two subtrees are of equal depth. The probability of having to update k nodes is 1/3^k. Rebalancing is O(1). Removing an element may imply more than one rebalancing (up to half the depth of the tree).
RB-trees are B-trees of order 4 represented as binary search trees. A 4-node in the B-tree results in two levels in the equivalent BST. In the worst case, all the nodes of the tree are 2-nodes, with only one chain of 3-nodes down to a leaf. That leaf will be at a distance of 2logn from the root.
Going down from the root to the insertion point, one has to change 4-nodes into 2-nodes, to make sure any insertion will not saturate a leaf. Coming back from the insertion, all these nodes have to be analysed to make sure they correctly represent 4-nodes. This can also be done going down in the tree. The global cost will be the same. There is no free lunch! Removing an element from the tree is of the same order.
All these trees require that nodes carry information on height, weight, color, etc. Only Splay trees are free from such additional info. But most people are afraid of Splay trees, because of the ramdomness of their structure!
Finally, trees can also carry weight information in the nodes, permitting weight balancing. Various schemes can be applied. One should rebalance when a subtree contains more than 3 times the number of elements of the other subtree. Rebalancing is again done either throuh a single or double rotation. This means a worst case of 2.4logn. One can get away with 2 times instead of 3, a much better ratio, but it may mean leaving a little less thant 1% of the subtrees unbalanced here and there. Tricky!
Which type of tree is the best? AVL for sure. They are the simplest to code, and have their worst height nearest to logn. For a tree of 1000000 elements, an AVL will be at most of height 29, a RB 40, and a weight balanced 36 or 50 depending on the ratio.
There are a lot of other variables: randomness, ratio of adds, deletes, searches, etc.
It is just the choice of your implementation - they could be implemented as any balanced tree. The various choices are all comparable with minor differences. Therefore any is as good as any.
Update 2017-06-14: webbertiger edit its answer after I commented. I should point out that its answer is now a lot better to my eyes. But I kept my answer just as additional information...
Due to the fact that I think first answer is wrong (correction: not both anymore) and the third has a wrong affirmation. I feel I had to clarify things...
The 2 most popular tree are AVL and Red Black (RB). The main difference lie in the utilization:
AVL : Better if ratio of consultation (read) is bigger than manipulation (modification). Memory foot print is a little less than RB (due to the bit required for coloring).
RB : Better in general cases where there is a balance between consultation (read) and manipulation (modification) or more modification over consultation. A slightly bigger memory footprint due to the storing of red-black flag.
The main difference come from the coloring. You do have less re-balance action in RB tree than AVL because the coloring enable you to sometimes skip or shorten re-balance actions which have a relative hi cost. Because of the coloring, RB tree also have higher level of nodes because it could accept red nodes between black ones (having the possibilities of ~2x more levels) making search (read) a little bit less efficient... but because it is a constant (2x), it stay in O(log n).
If you consider the performance hit for a modification of a tree (significative) VS the performance hit of consultation of a tree (almost insignificant), it become natural to prefer RB over AVL for a general case.
Consider a sequence of n positive real numbers, (ai), and its partial sum sequence, (si). Given a number x ∊ (0, sn], we have to find i such that si−1 < x ≤ si. Also we want to be able to change one of the ai’s without having to update all partial sums. Both can be done in O(log n) time by using a binary tree with the ai’s as leaf node values, and the values of the non-leaf nodes being the sum of the values of the respective children. If n is known and fixed, the tree doesn’t have to be self-balancing and can be stored efficiently in a linear array. Furthermore, if n is a power of two, only 2 n − 1 array elements are required. See Blue et al., Phys. Rev. E 51 (1995), pp. R867–R868 for an application. Given the genericity of the problem and the simplicity of the solution, I wonder whether this data structure has a specific name and whether there are existing implementations (preferably in C++). I’ve already implemented it myself, but writing data structures from scratch always seems like reinventing the wheel to me—I’d be surprised if nobody had done it before.
This is known as a finger tree in functional programming but apparently there are implementations in imperative languages. In the articles there is a link to a blog post explaining an implementation of this data structure in C# which could be useful to you.
Fenwick tree (aka Binary indexed tree) is a data structure that maintains a sequence of elements, and is able to compute cumulative sum of any range of consecutive elements in O(logn) time. Changing value of any single element needs O(logn) time as well.
As a follow up question related to my question regarding efficient way of storing huffman tree's I was wondering what would be the fastest and most efficient way of searching a binary tree (based on the Huffman coding output) and storing the path taken to a particular node.
This is what I currently have:
Add root node to queue
while queue is not empty, pop item off queue
check if it is what we are looking
yes:
Follow a head pointer back to the root node, while on each node we visit checking whether it is the left or right and making a note of it.
break out of the search
enqueue left, and right node
Since this is a Huffman tree, all of the entries that I am looking for will exist. The above is a breadth first search, which is considered the best for Huffman trees since items that are in the source more often are higher up in the tree to get better compression, however I can't figure out a good way to keep track of how we got to a particular node without backtracking using the head pointer I put in the node.
In this case, I am also getting all of the right/left paths in reverse order, for example, if we follow the head to the root, and we find out that from the root it is right, left, left, we get left, left, right. or 001 in binary, when what I am looking for is to get 100 in an efficient way.
Storing the path from root to the node as a separate value inside the node was also suggested, however this would break down if we ever had a tree that was larger than however many bits the variable we created for that purpose could hold, and at that point storing the data would also take up huge amounts of memory.
Create a dictionary of value -> bit-string, that would give you the fastest lookup.
If the values are a known size, you can probably get by with just an array of bit-strings and look up the values by their index.
If you're decoding Huffman-encoded data one bit at a time, your performance will be poor. As much as you'd like to avoid using lookup tables, that's the only way to go if you care about performance. The way Huffman codes are created, they are left-to-right unique and lend themselves perfectly to a fast table lookup.