A pseudoknot is a name applied to any RNA secondary structure (folding) that cannot be expressed in standard dot-bracket notation, i.e. with only '.', '(' and ')'.
Consider this example taken from Nando's blog post. To write this structure in dot-bracket notation, we start at the 5' end in the lower left corner. First there are six unpaired bases, so that would be 5 dots. Next come six paired bases, where this is the first time we have come to this pair. These are wriiten as open parenstheses. Then come 6 more unpaired bases, written as six more dots. At this point, we have
The next three bases are paired, and it is the first time we've visited these pairs, so we add three more open parentheses
The next six bases are also paired, but these are the second half of pairs we have already visited, so we would nortmally write them as closed parentheses, i.e.
But this isn't right, because the first three closing brackets would be interpreted as matching the closest previously unmatched brackets, and that would say we have a hairoin with an empty loop.
A common convention for writing pseudoknots is to use square brackets in adition to parentheses. With this convention, our example ccan be written
Here, open parentheses are only paired with closed parentheses and open brackets are only paired with closed brackets. This notation suffuces for most pseduknots seen in natural RNA. But from the perspective of mathematical possibilities, even more distinct symbols might be needed to have all the bases match up correctly.