If you really want to break free [1] from the von Neumann bottleneck [2] , grab yourself an FPGA starter kit [3] and start playing with Verilog or VHDL. FPGAs are quite interesting now that features like hardware [4] multipliers [5] are becoming more widespread.
Links:
[1] http://blogs.nubgames.com/code/?p=4
[2] http://www.stanford.edu/class/cs242/readings/backus.pdf
[3] http://www.altera.com/products/devkits/altera/kit-cyc2-2C20N.html
[4] http://www.xilinx.com/products/design_resources/dsp_central/grouping/fpgas4dsp.htm
[5] http://www.altera.com/products/devices/stratix2/overview/st2-overview.html