Store n:m data with most n:1 tuples
Using an intermediate table is the proper way to do this.  With proper indexes and good maintenance it won't hurt performance much at all.

Avoid the multi-column solution (refid1, refid2 etc.) - that's repeating groups and not properly normalised.  Plus your queries will degenerate into a dog's breakfast of OR clauses and CASE statements, which will hurt performance.

You may get some benefit from subtyping your data rows so most use your current arrangement but those with multiple reference use a different one.  Your queries will UNION ALL the two sets to get the total.  I really doubt the overhead from this will compensate for the mostly-redundant join to data_reference.

The subtyped schema would look like this:

create table data (
    id int identity not null,
    ref int not null,       -- set to a standard value for rows with &gt;1 reference.
    is_single_reference bit not null
);

create table reference (
    refid int not null
);

create table data_reference (
     dataid int not null,
     refid int not null
);


The query would be something like:

select
    &lt;whatever&gt;
from data as d
inner join reference as r
    on d.ref = r.refid
where d.is_single_reference = 1

union all

select
    &lt;whatever&gt;
from data as d
inner join data_reference as dr
    on dr.dataid= d.id
inner join reference as r
    on r.refid = dr.refid
where d.is_single_reference = 0;


This is fairly ugly.  You'd be lucky to avoid two scans of data.  Ideally you'd want filtered indexes but IIRC these aren't availble until SQL Server 2008.  The applicaiton would have to know how many reference rows there were per data, set flags accordingly and maintain those if the number changed.  Just use the n:m approach.