openusability.org has created and made public an online usability survey to collect data on the usability of the oxygen icon set. you can find the survey here.
to quote Riccardo Iaconelli of the kde artist team:
It takes just a few minutes to answer it, and your small effort can lead to a
much more usable KDE 4.
please participate to help make kde4 that much better =)
25 comments:
While I applaud everyone involved in trying to design better (for various definitions of better) icons and launching a survey... This particular survey is worse than unhelpful - at best, it will confirm or deny the surveyor's suspicions. If his/her suspicions are different from everyone else's preferences/needs, asking them which of the provided images looks like a PDF isn't going to help.
@lee:
> This particular survey is worse
> than unhelpful
but how do you really feel? ;-P
seriously though, given that this was done by people who actually do these things for a living (usability research and studies) i'm inclined to consider their efforts as having at least some founding in reality.
> asking them which of the
> provided images looks like a PDF
> isn't going to help.
there are two primary ways icons of these sort get used:
- the user is looking for a pdf and the icon helps them visually filter them
- the user is looking at a file, action or application and trying to determine what it is/does
the former is indeed easier to test for in an automated multiple choice survey.
however, many of the questions in the survey also address the latter.
is this type of survey "good enough" to serve as the only data point? nope. but taken in a wider aggregate of data which is then mined by someone who knows how to do such things, i think it's a valuable addition.
it's a lot like the "better desktop" project stuff. on its own it's not particularly useful to base decisions on, but collect enough of these different sets of data using different methodologies and contexts and one can start to get a pretty decent idea of what's going on.
Wow, are these the same people that brought us dolphin is more usable than konqi? If so, it all makes sense now. I agree totally with what lee is saying above.
Questions like: You want to make a backup copy of a CD-ROM. Which icon takes you to the CD recording application?
I don't want to go to a goddamn program, I want to right click and choose backup.
Hey Aaron,
I agree in part with the first poster: though this sort of survey is more than welcome, this particular one has some very important shortcomings. One example: none of the options given for "lock screen" was clear. But there was no way to say that! I chose that option based on what I already knew about Oxygen, and I reckon many others did the same. Now, judging based solely on these results for that question, the Oxygen guys might get the wrong impression that everybody finds their "lock screen" very intuitive!...
Anyway, I think having a new, more extensive survey about the Oxygen icons would be an excellent idea. I, for one, would gladly participate and encourage others to do so. However, please give us the option to at least add remarks to each question!
@anonymous:
"are these the same people that"
no, and -1 for pointless sniping.
i really need to get around to declaring my blog a positivity zone and just start deleting garbage like this. there is always a constructive way to say something, even when you don't agree with it, and for those who can't be bothered to try and do so i really don't have the time and energy to deal with them.
"I don't want to go to a goddamn program, I want to right click and choose backup."
of course. and that will still be there. however, that action:
a) needs an icon, and
b) isn't the only way people interact with the system
some people, believe it or not!, actually do look for the icon in their app menu.
remember this is about the clarity of purpose of the icon, not the actual method someone uses to get to that icon.
@dario:
"none of the options given for "lock screen" was clear. But there was no way to say that! I chose that option based on what I already knew about Oxygen, and I reckon many others did the same."
given that the user will be presented with a set of icons to choose from (they are already at the log out dialog) this question is less absurd than it seems. that said, it would indeed be nice to have a "i have no friggin' clue" option.
i'll suggest it to the surveyors.
having full text on each question might result in some very high volumes of data and just overwhelm them when it comes time to collation. there is, iirc, a comment box at the end? (i went through it a week or so ago, so perhaps my memory is failing me there).
i'd also expect that if the icon isn't clear that there will be a high miss rate with people selecting the other icons. given that not many people are familiar with oxygen, as you apparently are, that's not an unreasonable expectation.
I think the artists should decide which icons they choose, because they're more competent.
The end of the survey contains a typo, I believe they misspell oxygen as oxigen or something similar (looks like it might be a translation error?).
I'm going to have three more people run through it tonight, two who are Windows only users, and one who "grew up" on Apple machines, and just recently flipped to KDE (and whom I will marry in about 40 days, although the KDE part wasn't the reason).
And I still wondering why the print preview icon is an electrocuted printer.
The survey was good but those file icons were really small. I am using just 1152x864 but had to zoom into those icons to make something out of them. I wonder what would have happened with higher display resolution.
Also, I agree about that lock the computer icon. Simply couldn't differentiate.
> Also, i agree about the lock the
> computer icon. Simply couldn't
> differentiate.
I may agree with that, but there is a trick here.
You must read the all text ( lineal thinking ) for understand what the icon means ( non lineal thinking ).
So if you don't read the full text, you can not get what the icon means.
And the color of icons (black) don't helps here. =(
Thanks for the feedback, everybody.
About the third “don't know/none of these“ option many of you would like: If we had this, we wouldn't know why people didn't pick any one, or which they find particularly bad or which could work with some improvements. The only way to find that out is by using free-text comment boxes for every single question. Which is a great way to collect what everybody has to say and is thinking - for a small number of people (say, in a live usability test). But if a survey with a number of these boxes has 6000+ respondents (and more coming in all the time) it is very time-consuming and difficult to analyze the results ((6000 x number of boxes per survey)/2 people that will have to read, interpret & classify each and every sentence). The numerical results we get from the regular answers can be crunched by the computer with a few formulas, on the other hand. Still, we are thinking of a way to compromise on this for next time. Changing anything on the running survey is not possible, unfortunately.
That said, this survey is not the only method we are using to test Oxygen.
Tina
@Tina Trillitzsch
So because you have too few resources to read text responses you choose to force people to take a guess, which may or may not contaminate the data, without leaving you a way to be sure about how it affect the result.
I think giving the users the "don't know" choice would be far superior, even if you don't know why they didn't choose an icon, right now you don't even know if they made a qualified choice or if they just guessed because they had to.
I actually had a fair amount where i had to guess, especially the mime types, the icons were so small and blurry that i really couldn't see what many of them were depicting. The uniform colors sure didn't help.
The shutdown icons test was very "helpful", i could guess the meaning of all the icons, simply by deduction because you listed all the possibilities. Now if i had to guess what each of those icons did, without having a list of choices, i wouldn't be able to guess all of them.
While I also applaud the attempt... IMHO this survey is next to useless, or even worse than that.
It is indeed something designed to confirm the surveyor opinions and nothing else.
I *regret* taking it, as someone will use my answers to justify his/her own mistaken choices.
[...]
The mini-icons for documents were WAY too small for me. But, no, no way to choose a "I can't read any of these icons" option.
No where to say: ""there is not enough difference between all these icons for me to tell them appart.""
Never options like: "I don't know which of these icons does that".
[...]
Where can I go to explain the specific problems I have with KDE icons to the Oxygen people? Is this the right place?
hey aaron, how is plasma going? when are we going to see a preview or something of that? we are really excited here about plasma, how is going to look and when we are going to see the first things about plasma, are you actually working on it?
pd, this oxygen icon usability study is awesome! just completed it!
@troels
"The shutdown icons test was very "helpful", i could guess the meaning of all the icons, simply by deduction because you listed all the possibilities. Now if i had to guess what each of those icons did, without having a list of choices, i wouldn't be able to guess all of them."
We presented the icons together as a group because that's how you will encounter them in the real environment. There you will base your choice on the surrounding icons as well. Consequently we are evaluating with a rather high cut-off rate for this specific question.
@lee (and francisco, who said something similar):
"at best, it will confirm or deny the surveyor's suspicions. If his/her suspicions are different from everyone else's preferences/needs"
That was the intention, indeed. What makes you think the suspicions of people experienced in interface evaluation and user testing are worthless?
@evan:
"Oxicon" instead of the more obvious Oxygen is intentional and not a typo.
The reactions here and in various other places prompt me to make it even more clear that this survey is not a test of the users/respondents/you, but a test of the icons (with the help of users). You can't pick a "wrong" icon, and you cannot do "badly".
Also, some of you appear to believe that we just throw a number of random icons at the respondents and have them pick one that we then immediately use as the new icon. Instead, we are giving them the one icon that was specially designed and carefully chosen for the purpose and a number of alternatives (in most cases). If people pick any other icon than the one that was specifically designed for that feature, it tells us a lot about potential improvements by looking at the one that _was_ picked instead.
Tina
@tina
Sorry, it sounds very complex to me.
The lock session icon
(-) is uggly and very cryptic.
Another thing to note: the icon designers and the survey designers are independent - and that's exactly how it should be. None of the people creating the survey created any of the icons and vice versa.
Tina
@tina:
That was the intention, indeed. What makes you think the suspicions of people experienced in interface evaluation and user testing are worthless?
1. There is a very fundamental difference between performing an experiment to check an hypothesis while giving it reasonable chances to fail, and therefore allowing the hypothesis to be shown wrong. And designing a biased experiment to confirm an hypothesis regardless of its merit.
This survey seems, to me at least, be more like the later kind. Since there was no way enter negative data.
2. The opinion of your experienced in design people (whatever that means) is not useless. But isn't the point of asking users to answer a survey to, well, survey what the users opinion is?
[...]
Options I would suggest you to add next time you run a survey (if you are not going to change much):
You want to write to a CD, you see this N icons in a menu:
[ ] A
[ ] B
etc
[ ] None of the icons seems to do that, I would go looking into another menu.
===========
Which of these documents are spreadsheets?
[]tiger
[]zebra
[] I can't see any spreadsheet icon there.
[] I can't tell the icons appart from one another.
If the user clicks on any of the last two, you should ask:
Was the icon size adequate?
yes
too big
too small
---
Did the icons have enough color contrast? yes/no.
=============
Well, I better stop... have a good night.
Cheers,
Francisco
I agree with the last comment.
At some page, if there is no text "telling me" these buttons are for: .....
I was looking for:
[x] I don't know
[x] These don't make any sense
[x] How am I suppose to know
[x] How is this suppose to be usable?
[x] I have an engineering degree and I'm trying to guess... hmmmm
The point is:
If you "really try hard" and by "mistake" you click the "right" answer, then the survey will say:
"hey this icon is perfect, when it needs some work".
e.g.: [-] is lock, a key would be much better.
That being said, I have nothing against you aaron, I think you do a great job, it's just that I feel like some other that the survey is "biased" and need other options, that's all.
I haven't used KDE in a while and I found some of these icons confusing as an outsider.
For example as stated above I had no idea what the lock icon was. The poweroff one was another vague one.
The document icons were too small and too white. I really couldn't tell what was what by the tiny features. I didn't check anything off for that section, hopefully that brought my point across.
I'm hopeful this survey will show developers which icons users found confusing and we'll see changes based off them.
Also question for you guys.. why are the back and forward arrows black? They seem off place and thin. Why not use something thicker and more colorful?
WOW
Those Office icons were worse than bad ....
What about enhancing Monochrome icon set ? (MonoOxy anyone ?)
I am posting this here as well in case anyone reading this blog entry wonders why the survey isn't active anymore: The survey was scheduled to run one week and is now closed.
So thanks everyone for participating.
Florian
yeah i didn't like those office icons, the rest was ok
I just want to say thanks to everybody that took the time to take the survey. IM sure the results will tel us alot on the uzer preferences.
Some notes....
The office icons are being redone at this moment the new ones look alot beter.
The lock/switch/power icon were suposed to cnage in a long time I was under the impresion they were diferent enough to ick them crectly but we needed some kinda of test to get the anser.
The way the question is posed is good, its exctly how it will apera on a regular use of the desktop all those icons are shown toguether and with those 3 options. all i wanted was some kind of visual continuaty.
BTW check the current icons kde is using the switch uzer is .... lot more confusing IMO.
But i gess i got much more than what i bargain for.
People relax, please!
Suposed to have fun, i have lost of fun doing this icons.
Post a Comment