Setter ignored on collection

I remember reading somthing about the behaviour exhibetet by the
program below (where a private collection is altered without going
through its accessor). But cannot seam to find the info anymore - can
someone enlightenme as to why this works:

Class for test
#################
public class TestMe {
    private Set<String> values = new HashSet<String>();
    private int called = 0;
    public Set<String> getValues() {
        return values;
    }
    public void setValues(Set<String> values) {
        for (String v : values) {
            if (v.equals("invalid"))
                throw new IllegalArgumentException("invalid value");
        }
        this.values = values;
        called++;
    }
    public int getCalled() {
        return this.called;
    }
}
#################

Test code
#################
TestMe test = new TestMe();
String valid = "valid";
String anotherValid = "anotherValid";
String invalid = "invalid";

// sets values without calling setter
test.getValues().add(valid);
if (test.getValues().contains(valid)) {
    System.out.println("setter is called: " + test.getCalled()
            + " times, values are: " + test.getValues().toString());
}

// sets values by calling setter
Set<String> values = new HashSet<String>();
values.add(anotherValid);
test.setValues(values);
if (test.getValues().contains(anotherValid)) {
    System.out.println("setter is called: " + test.getCalled()
            + " times, values are: " + test.getValues().toString());
}

// does not work as expected
test.getValues().add(invalid);
if (test.getValues().contains(invalid)) {
    System.out.println("setter is called: " + test.getCalled()
            + " times, values are: " + test.getValues().toString());
}

// works as expected
test.getValues().remove(invalid);
values.add(invalid);
boolean caughtException = false;
try {
    test.setValues(values);
} catch (IllegalArgumentException e) {
    caughtException = true;
}
System.out.println("caught exception: " + caughtException);
#################

The output becomes
#################
setter is called: 0 times, values are: [valid]
setter is called: 1 times, values are: [anotherValid]
setter is called: 1 times, values are: [invalid, anotherValid]
caught exception: true
#################

I can see why this might work given the reference to the collection
returned from the getter, but is there a way to stop this from
happening (besides declaring the collection as unmodifiable ?)