how about you give me the name model of you Hi-Fi ?
Generally for stuff like that to work best theres usually 2 ports on the Hi-Fi its self matching your colour.
there would be setting on your hi-fi called AUX (or close to it)
So you would have
CD,Tape,Radio,AUX. I use to have my PC connected up to speakers on TV and the speakers on Hi-Fi lol
Very strange configuration but was funny because it like played in a row E.G during a game.
PC's Speakers - gun shot bang .
then the rest like followed in order bang bang bang bang bang etc.
was funny.
you have the PC side set correctly just a matter of finding the correct spot on hi-fi.
if both speakers aren't the same it means you don't have the right connections.
Red and white connector, like Left and Right speaker sort of.
so if you switch those round does the one speaker still sound the same? if it does theres a problem with were you pluging the other piece of the cable.
example of ports your need on back of you Hi-Fi
Should be colour coded