The system is simplicity itself. Each user of the system makes two numbers, eU and nU public and keeps a number dU secret. In order for A to send a message to B, A looks up B's public values and, if the message is m (written as a number), then A blocks the message into pieces of size < nB and sends c = meB mod nB. Then B decodes by m = cdB mod nB. The security of the system lies in the choices of the public and private keys. To understand these choices we need to consider some number theory.
For any integer n, Euler's Totient Function,
(n) is the number of integers greater than or
equal to 1 which are relatively prime to n. It can be shown that:

(12) = |{1,5,7,11}| = 4.12(1-1/2)(1-1/3) = 12(1/2)(2/3) = 2(2) = 4.
Euler's Theorem: If gcd (a,n) = 1 then 
Corollary: If n is a product of distinct primes then
for any integer t.
Pf: Let p be any prime that divides n. If gcd(a,p) = 1, then
is valid by
Euler's Theorem. On the other hand, if a
0 mod p, then the statement is trivially true. Since
the congruence holds for each prime dividing n, it also holds for n.
For the RSA choices, each user selects two prime numbers (about 100 digits long) p and q
and sets nU = pq. Note that
(nU) = (p-1)(q-1). [p and q are no longer used, but must be kept
secret]. Next, eU is selected subject to 1 < eU <
(nU) and gcd(eU,
(nU)) = 1. Finally, dU is
calculated (using the extended Euclidean
Algorithm) so that eU dU
1 mod
(nU). We now see that

Finding p and q can be done with a fast primality tester.
The practical user of RSA must be on guard against some common pitfalls, known as protocol failures. In these cases, how a message gets encoded to a numerical equivalent may defeat the cryptosystem.
The RSA scheme can be used for signatures in the usual way.
The only known way to break the system is to find
(nU) which is almost equivalent to
factoring nU. The Rabin variation is a version of RSA in which it can be shown that the security is
equivalent to the difficulty of factoring.
So the security rests (perhaps) on the difficulty of factoring large numbers. To avoid those situations where fast factoring algorithms exist one should select p and q so that
Phil Zimmerman's public domain program PGP (Pretty Good Privacy) is a combination of RSA and a fast private key system, IDEA (International Data Encryption Algorithm). The RSA algorithm is used to encrypt the private key generated for the IDEA. Once this is transmitted, the private key is used to decrypt the message which is sent, encrypted by IDEA. IDEA is considered to be much stronger than DES and uses a 128 bit key.