Even though C# and Db are represented on a piano (as well as other instruments) by the same key, theoretically speaking, they are not the same note. Db does not appear in the diatonic key of E major. C# is the only relative minor of E major. The interval between E and C# is a major 6th while the interval between E and Db is a diminished 7th.
But yes, the key signature does remain for any key and its relative minor. But 4 flats is the key of Ab/Fm while 4 sharps is the key of E/C#m. Following the rules for the construction of a minor key (TSTTSTT), The key of Db minor would be Db (tonic), plus a whole step gives us Eb (supertonic), plus a half step gives us Fb (median), plus a whole step gives us Gb (subdominant), plus a whole tone gives us Ab (dominant), plus a half step gives us Bbb (submedian), plus a whole step gives us Cb (subtonic), plus a whole step gives us Db (repeated tonic), or 8 flats (since B is double-flatted it counts as 2 flats).
Theoretically, we could extend the circle of 5ths indefinitely with large qualtities of sharps and flats, but through convention (and common sense), we only use 15 key signatures: 7 flats, thru 7 sharps plus the empty key signature (no sharps and no flats). The circle of 5ths on this site only shows 12 which is common for pop/rock/jazz, but in the realm of classical music you will find all 15 key signatures used.