I'm not sure I understood correctly your use case. But, if you have two
processes/threads randomly switching O_NONBLOCK on/off, your problems
arise not only the F_SETFL time.
If one of the tasks is not expecting an fd to be O_NONBLOCK, that will
likely end up not handling correctly read/write-miss situations.
In that case it'd be better to keep the fd as O_NONBLOCK, and manually
create blocking behaviour (when needed) with poll+read/write.
- Davide
-