Not sure what you are looking for here, if that respondent has answered only once per data source you can filter it based on data source which would solve your issue. 

​@Deepak sorry if I was not clear. 
Yeah, I get that part, but this is still not what I want.
If person A has responded in 5 surveys, and person B has responded in 4 surveys, Then I would like a count in the dashboard that says 2. Meaning it should count each person only once if they have given at least one response, no matter how many responses they have given.

​@JesperAndersen have you tried using the respondent funnel as data source? and then add that survey in widget you will get unique counts for that data source after which if you total that you will get unique respondents total.

​@Deepak not a bad idea, but I need this to work in an EX directory, and as far as I see, the respondent funnel is only possible to use in CX type directories.