The opposite end of the spectrum from gothy and Victorian is colorful girlishness. Think all things twee with a retro bent. Barrettes, floral details and candy-inspired nail polish are great details to pull off the look.
Also not dark, academia. (Unless dark libraries count.) Velvet bottoms would look great with argyle, a colorful blazer or cardigan and some form of oxford.
I own two black velvet blazers (and one burgundy velvet blazer), and I'd like to think I know how to style them. In the fall, I usually wear one with a colorful shirt, colorful pumps, jeans and great vintage jewelry.
Another great use for a black velvet blazer is extending the season of a print dress. The example below uses an animal print, but any print dress gets an air of refinement with a black velvet blazer.
For some reason, when it comes to black velvet, people want to start wearing it with lace, ruffles and other romantic notions. But remember, it's just black.