Regardless of the name, penny stocks cost a lot more than a penny. On a large stock exchange, a penny stock is anything that sells for less than $5. But this is just one definition, and there are many.
Penny stocks are usually stocks with a market value less than $200 million (this number can vary.) The most obvious definition is any stock with a price less than $1 per share.
Penny stocks are considered high risk because they are stocks for low valued companies and they do not need to have independently audited reports. They also aren't analyzed by stock analysts because there is little interest.
If you do wish to trade in penny stocks then the broker is obliged to inform you of the risk involved.
So, unless you know what you are doing, penny stocks, while sounding like a cheap investment could end up costing you a bundle.